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Executive  Summary 


1.  Introduction 

The  challenge  for  most  artificial  systems  that  operate  autonomously  in  the  real  world 
environment  is  how  to  cope  with  dynamic  environment  with  limited,  uncertain,  and  noisy 
information.  Artificial  intelligence  and  intelligent  robotics  research  has  been  trying  to  solve 
such  a  problem  by  either  improving  accuracy  of  recognition  systems  or  by  integrating  multiple 
source  of  information.  In  addition,  architectural  issues  has  been  discussed  on  whether  classical 
Sense-Model-Plan-Act  architecture  or  the  subsumption  architecture  better  suits  for  autonomous 
agents.  Information  fusion  issue  is  tightly  coupled  with  behavioural  control  as  overall 
performance  of  the  autonomous  system  is  the  ultimate  concern. 

The  work  performed  focused  on  identifying  possible  system  architecture  for  realistic 
information  fusion  and  corresponding  reactions  under  uncertain  environments.  Our  research 
starts  from  analysing  issues  in  existing  paradigm  of  autonomous  agent  and  AI  architectures, 
redefine  needs,  and  propose  a  suitable  architecture.  In  this  research,  it  was  essential  to  learn 
from  biological  systems  where  various  species  has  evolved  to  adapt  to  uncertain  and  dynamic 
environment  for  survival. 

2.  Limitations  of  Current  Architectures 

2.1.  Robustness  of  Sensing  Systems 

There  are  several  approaches  proposed  to  provide  intelligent  and  autonomous  behaviours 
interacting  with  real-world  environment.  In  this  article,  “real-world  environment”  refers  to 
physical  real  world  with  open  dynamics  that  is,  in  general,  outside  of  the  laboratory  space.  In 
such  environment,  numbers  of  unexpected  incidents  may  take  place  that  are  not  expected  at  the 
time  of  system  design.  One  of  the  problems  for  man-made  systems  is  its  inherent  fragility 
against  unexpected  perturbations  that  are  not  designed  for.  Major  efforts  have  been  made  to 
improve  accuracy  of  recognition  for  vision  and  audition,  but  mostly  for  a  single  modality  of 
perception.  For  example,  vision  systems  are  often  designed  to  recognize  object  based  on  shape 
of  objects  based  on  image  captured  by  the  camera.  Some  other  systems  focus  on  color  or 
pattern  matching  with  stored  images.  It  was  relatively  recent  that  efforts  are  made  to  integrate 
information  from  multiple  sources,  and  such  efforts  are  made  only  between  two  or  three  types 
of  modalities.  Our  previous  work  discovered  that  accuracy  of  each  modality  of  perception  can 


be  improved  when  information  in  each  channel  is  properly  integrated  [Nakagawa,  et  al,  1999; 
Nakadai,  et  al,  2000].  However,  the  integration  has  only  been  taken  place  between  color-based 
labelling  vision  system  with  physical  a  priori  knowledge  and  auditory  scene  analysis  system. 
The  problem  of  these  approaches  is  that  they  may  not  be  sufficiently  robust  against 
perturbations  of  signals  that  provide  key  to  recognition  of  situation  the  agent  is  located. 
Shape-based  visual  recognition  system  would  not  work  when  a  part  of  object  is  not  visible  from 
the  camera.  Color-based  recognition  cannot  work  when  light  level  is  very  low  for  color-based 
recognition.  While  integration  is  effective,  integration  of  small  numbers  of  different  modalities 
is  not  sufficiently  robust  against  perturbations.  Since  the  system  will  be  used  under  hostile  and 
uncertain  environments,  robustness  shah  be  the  major  concern  in  the  architectural  design  of  the 
system.  Consideration  shah  be  given  at  ah  levels  of  system  to  enhance  its  robustness.  Our 
preliminary  work  clearly  suggested  that  increase  of  modalities  significantly  improve  not  only 
overall  systems  performance  on  recognition  of  the  situation,  but  also  accuracy  of  recognition 
within  each  sensory  channel  [Okuno,  et  al.,  2004]. 

2.2:  Contextual-thickness  of  Artificial  and  Biological  Systems 

Other  significant  issue  for  autonomous  system  is  their  high-degree  of  dependence  for  context 
and  task  assigned.  Artificial  systems  are  generally  designed  for  specific  tasks,  and  can  be 
extremely  effective.  The  champion  example  is  computer  chess  which  Deep  Blue  beats  Gary 
Kasparov,  the  human  world  champion.  However,  deep  blue  will  not  be  able  to  drive  a  car,  clean 
houses,  sing  a  song,  etc.  Its  function  is  narrowly  tuned.  On  the  other  hand,  biological  systems, 
typically  human  being,  can  cope  with  broad  tasks  at  certain  level  of  competence.  As  shown  in 
Fig  1,  biological  systems  are  “contextuahy-thick”  and  artificial  systems  are  not  [Kitano  et  al., 
1993]. 


Performance 


2.3  Architecture  for  Intelligent  Systems 

In  addition,  there  has  been  an  argument  that  traditional  Sense -Model-Plan-Act  (SMPA) 
architecture  (Fig2a)  is  not  suitable  for  autonomous  system  in  the  real-world,  and  alternative 
architecture  called  “the  subsumption  architecture”  has  been  proposed  by  Rodney  Brooks 
[Brooks,  1991].  The  subsumption  architecture  is  one  of  example  where  attempt  has  been  made 
to  learn  from  biology  as  it  was  inspired  by  insect  behaviours  during  Rod  Brooks’s  stay  in  a 
village  in  Thailand.  The  subsumptoin  architecture  is  an  attempt  to  use  the  physical  real-world 
as  a  model  itself  without  having  a  model  of  the  world  within  the  autonomous  system.  The  idea 
is  to  hardwire  sense-reaction  circuits  for  low-level  process  and  add  layers  of  such  processes 
with  mostly  inhibitory  control  (Fig  2b).  It  has  been  very  successfril  for  simple  tasks,  but  not 
shown  to  be  scalable  for  more  complex  tasks.  In  addition,  it  was  not  sufficiently  effective  in 
coping  with  multiple  tasks  and  dynamical  environment  where  unexpected  perturbations  are 
imposed.  Each  elementary  behaviours  are  hardwired  which  assume  a  certain  stimuli  shall 
trigger  a  specific  sequence  of  actions.  Thus,  such  system  lacks  robustness  against 
environmental  changes. 

These  arguments  on  architecture  of  intelligence  systems  miss  the  point  of  robustness, 
evolvability  and  global  properties  of  complex  networked  systems.  In  the  author’s  view,  these 
are  the  central  issues  that  guides  us  define  suitable  architecture  for  intelligence  system  under 


uncertainty  which  is  best  investigated  in  biological  systems  that  have  actually  evolved  to 
survive  under  uncertain  and  dynamic  environment. 


(A)  Sense-Model-Plan-Act  (SMPA)  architecture 


(B)  Behaviour-based  AI  architecture 

Fig  2:  Architectures  for  Intelligent  Systems 


3.  Robustness  is  the  Fundamental  Issue 

Robustness  is  a  property  of  the  system  that  can  maintain  certain  properties  against  certain 
perturbations.  Such  properties  can  be  observed  for  both  artificial  and  biological  system  to 
different  degree.  However,  artificial  systems  are  nowhere  near  the  level  of  robustness  realized 
in  most  of  biological  systems.  This  is  a  strong  motivation  for  us  to  closely  look  into  robustness 
of  biological  system  so  that  we  can  learn  architectural  features  of  robust  systems  with  the  hope 
of  extracting  essential  and  universal  features. 

3.1  Basic  Mechanisms  for  Robust  Systems 

The  system  generally  exhibits  adaptation  against  changing  environments  and  insensitivity  to 
parametric  changes.  Such  properties  can  be  achieved  by  few  basic  mechanisms  that  are;  (1) 
feedback  control,  (2)  redundancy,  (3)  modularity,  and  (4)  structural  stability  [Kitano,  2004a]. 

Combination  of  such  basic  mechanisms  enables  us  to  design  systems  that  have  a  certain  level  of 
robustness,  such  as  airplane.  For  example,  airplane  is  a  system  that  is  designed  to  achieve  stable 


flight  against  taburance  of  atmospheric  conditions  and  possible  damages  of  airplane  subsystems 
A  flight  control  system  can  continue  to  maintain  its  ability  to  control  flaps  and  ladders  even  if 
one  of  four  hyduralic  systems  is  damaged.  This  is  possible  because  hyduralic  systems  are  built 
redundant.  Automatic  flight  control  enables  stable  flight  against  various  atmospheric 
perturbations  due  to  its  feedback  control.  In  addition,  it  is  composed  of  three  independent  flight 
control  computers.  Three  computers  are  developed  to  attain  identical  functions  (homogeneous 
redundancy),  but  designed  as  differently  as  possible  (heterogeneous  redundancy)  to  avoid 
common  mode  failure.  The  use  of  digital  system  decouples  functional  signal  layer  from  the  low 
level  voltage  fluctuation.  (Fig.  3) 
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In  principle,  robustness  of  the  system  can  be  enhanced  by  designing  the  system,  composed  of 
modular  elements,  to  have  redundancy  in  various  aspects  and  impose  appropriate  feedback 
control. 

However,  simply  combining  basic  mechanisms  does  not  archive  robustness  for  broader  range 
of  perturbations.  Airplane  assumes  most  sensory  readings  are  correct  and  sufficient  for 
autopilot.  It  is  effective  only  under  specific  assumption  where  input  information  channel  can  be 


pre-defined  and  systematically  integrated.  The  architecture  for  integrating  information  has  to  be 
defined  for  building  truly  robust  system. 

3.2  Global  Network  Architecture 

The  Bow-Tie  architecture  was  recently  proposed  as  a  global  network  structure  found  in  internet 
and  biological  systems.  It  consists  of  variety  of  inputs,  a  conserved  core  network,  output 
networks,  and  control  loops  spanning  within  and  between  each  sub-network.  (Fig  4) 


Figure  3 

n 

ln|Klt  pDRYSV 
<vlnsalin)t  uBd 
nulrktit«) 

3 

•:3  ^  t 

g  %  ^ 

Input  pnH.^-%% 
tvinjtulinit  and 
nulrkutt) 

£  £  ^ 

Input  priMTi’^^ 
(ikisulkix  iuid 
nulrktit«^ 

>  > 

a  £ 

InfHit  pnKt'^^ 
(tinsiilins  tuid 
nuirktitt^ 

Sysif  ni  Conlrol 


InU-rfaicv^  Ihiluiitii 


Output 

trv^puDM'v  and 
pru<iui‘t»l 


Output  prvci'vs 
4r\-\punur«  and 
pruduct^l 


Output  pruci'^v 
trx-\punM.'v  and 
pruducr*) 


Output  prvrvvi 
trr«punM.‘«  and 
prudui'tnl 


Fig  4:  The  bow-tie  architecture 


We  have  investigated  actual  biological  networks  on  signal  transduction  where  diverse  input 
stimuli  are  transmitted  into  nucleus  and  trigger  various  cellular  responses.  Comprehensive 
maps  of  EGFR  signalling  network  and  TLR  networks  are  created  for  the  first  time  [Oda  et  ah, 
2005;  Oda  and  Kitano,  2006].  The  structure  analysis  of  the  network  revealed  that  the  bow-tie 
structure  is  consistent  framework  for  network  involved  in  biological  signal  integration.  Diverse 
inputs  are  once  converged  into  smaller  numbers  of  components  in  the  middle  layer,  then 
relayed  to  output  after  a  certain  processing.  With  this  architecture,  it  is  expected  that  the  middle 
layer  may  have  a  certain  abstraction  and  classification  function  that  have  emerged  through 
evolution  (Fig  5).  This  resembles  the  three-layer  neural  networks  with  limited  hidden  layer  for 
generalization  capability. 
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Fig  5:  The  Classifier  Hyperspace  in  biological  signalling  systems 


4.  Implications  for  Robust  Engineering  Systems 

These  findings,  along  with  other  findings  not  listed  above,  has  interesting  implication  for 
designing  of  engineering  systems  that  are  robust  against  dynamic  change  of  environment. 

The  central  issue  is  to  consider  “robustness”  as  the  central  feature  of  engineering  system  design 
that  reflects  global  architecture  and  basic  mechanisms  described. 

4.1  The  Bow-tie  Architecture  for  Autonomous  Intelligent  Systems 

While  information  fusion  has  been  discussed  in  various  aspects  in  the  past,  the  issue  has  not 
been  settled  on  at  which  layer  the  information  shall  be  fused  and  what  is  suitable  architecture 
for  information  fusion.  Some  argued  for  low-level,  at  pixel  and  sensory  readout  level  [fkeda  et 
ah,  2003],  whereas  other  argues  for  fusion  at  concept  formation  level  at  symbolic  layer 
[Brachman,  et  ah,  1985].  Architectures  of  biological  networks  indicate  that  bow-tie  structure 
has  been  evolved  through  long  history  of  evolution  for  information  fusion  in  biological  systems 
with  only  a  handful  of  variation  at  the  global  level.  For  relatively  well-defined  set  of  stimuli,  a 
bow-tie  structure  that  has  limited  intermediate  nodes  can  be  very  effective  and  such  layer  shall 
be  evolved  or  adapted  for  generalization  and  concept  formation  through  stimuli  exposed.  In  the 
relatively  new  and  highly  diverse  stimuli,  specific  modification  to  downstream  of  bow-tie 
seems  to  be  the  proper  architecture  as  abstractions  may  not  sufficiently  evolved  or  requires 
more  precise  control  based  on  different  stimuli,  ft  is  yet  to  shown  theoretically  how  these  two 
types  of  global  architecture  may  affect  generalization  and  specific  behavioural  control  of  the 
system. 
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Fig  6:  System  Architecture  for  Stanly,  a  winning  autonomous  system  for  DARPA  Grand 

Challenge 

It  is  interesting  to  note  that  system  architecture  of  STANLY,  a  winning  autonomous  vehicle  for 
DARPA  Grand  Challenge,  resembles  a  partial  bow-tie  structure  where  diverse  inputs  converge 
into  smaller  numbers  of  modules  that  ultimately  control  throttle  and  steering  (Fig.  6)[Thrun,  et 
ah,  2006].  While  such  architecture  is  inevitable  when  only  two  degree  of  freedom  exists  for 
control,  it  is  the  first  step  toward  more  complex  systems  that  can  operate  under  different 
context.  In  the  Stanly  architecture,  output  network  do  not  exists  in  reality  as  possible 
behavioural  patterns  and  context  is  limited.  However,  if  the  system  must  operate  in  more 
complex  environment  with  higher  degree  of  freedom  for  control  as  well  as  required  to  perform 
in  multiple  contexts,  the  system  must  incorporate  mechanisms  for  coordinating  different  type  of 
behaviours. 

Nevertheless,  there  are  numbers  of  features  present  in  biological  system  are  yet  to  be 
implemented  in  engineering  system.  Although  not  all  of  them  are  suitable  for  artificial  system, 
there  are  numbers  of  control  architecture  that  are  worth  considering.  Some  of  the  work 
performance  under  the  grand  proposes  possible  bow-tie  architecture  and  multi  modal  sensory 
systems  [Kitano,  2004b;  Kitano  et  ah,  2004]. 


4.2  Layered  Control,  Context-Switching,  and  Evolution  of  Regulatory  Control 


Cells  operate  in  both  dynamically  changing  environments  as  well  as  in  rather  stationary 
environment.  During  the  developmental  stage,  cell  has  to  operate  in  radically  different 
environment  and  cell  itself  will  undergo  differentiation  to  make  major  switch  in  their  control 
mechanism  and  adapt  to  new  operating  context.  The  differentiation  often  took  place  with 
chromatin  structure  modifications  in  which  numbers  of  genes  are  affected.  Chromatin  structure 
change  is  true  program  of  the  cell  where  expression  of  genes  may  be  considered  as  microcode. 
Hierarchical  organization  of  program  execution  in  the  context-dependent  manner  is  one  of 
mechanism  that  can  robustly  cope  with  perturbations  in  different  context.  One  the  other  hand, 
bacteria  generally  does  not  perform  differentiation,  and  does  not  seem  to  have  major  chromatin 
remodelling  activity.  It  is  insightful  that  context-switch  mechanism  has  been  evolved  in  rather 
explicit  manner  in  multi-cellular  organisms,  but  not  in  other  organisms. 

Multiple  layers  of  control  are  yet  another  feature  that  are  typical  in  biological  systems  but  not 
clearly  observed  in  most  engineered  system.  Control  of  engineered  systems  tends  to  be 
mono-layer  due  to  transparencies  of  design.  Regulation  of  cellular  activities  are  modulated  by 
protein-protein  interaction,  ion  channel,  and  other  reactions  with  relatively  fast  temporal 
window,  transcriptional  and  translation  regulation  that  has  longer  temporal  window,  and 
chromatin  remodelling  that  act  as  contextual  switch  of  overall  genetic  program.  In  addition, 
significant  RNA  regulation  has  been  found  in  recent  days.  This  is  a  striking  discovery  where 
chromosome  regions  mostly  considered  as  “junk”  seems  to  function  to  regulate  cellular 
behaviour  as  noncoding  RNA  (ncRNA),  RNA  that  does  not  encode  protein  structure 
information  [FANTOM3,  2005].  Looking  at  the  evolution  of  organisms,  it  is  interesting  to  note 
that  components  involved  in  cellular  regulatory  mechanism  is  increasing  at  accelerated  speed, 
much  faster  than  the  speed  of  genome  size  increase.  This  obviously  means  biological  system 
with  higher-level  of  complexity  are  required  to  have  extensive  and  multilayered  control 
mechanisms.  In  fact,  it  may  be  the  case,  what  may  limit  the  evolution  of  organisms  in  terms  of 
complexity  is  difficulty  to  accommodate  more  sophisticated  control.  ncRNA  is  rarely  observed 
in  simple  organisms,  and  yeast  is  the  simplest  eukaryotic  cell  that  staert  to  have  ncRNA  based 
regulation.  In  mammalian,  it  was  found  that  most  transcripts  are  ncRNA.  The  ratio  of  ncRNA 
increases  as  complexity  of  organism  increase.  It  is  yet  to  be  determined  how  ncRNA  contribute 
to  robustness  of  cellular  system,  particularly  in  integrating  information  and  ensure  robust 
response  of  the  cell.  However,  discovery  in  this  aspect  of  cellular  regulation  may  provide  us 
novel  insight  on  how  complex  regulation  have  evolved  in  biological  systems,  and  how  to 
re-implement  such  scheme  in  engineered  system  that  will  be  increasingly  complex  in  future. 


Implications  and  how  to  implement  this  context  switch  and  evolution  of  layers  of  regulatory 
control  in  engineering  systems  is  yet  to  be  found,  however,  the  work  performed  in  the  project 
have  identified  that  this  is  potentially  an  important  technical  feature  to  build  truly  autonomous 
systems. 

0- 

i  ,11 


l»a5o- 


Natur*  Raviaws  |  Oanatfca 


Nature  Reviews  |  Genetics 

Fig  7:  Evolution  of  regulatory  control  in  biological  systems 
(Mattick,  RNA  REGULATION:  A  NEW  GENETICS?  Nature  Reviews  Genetics  5,  316-323 

(2004);  doi:10.1038/nrgl321) 


4.  3  Robustness  Trade-offs 

Trade-offs  inherent  in  evolvable  robust  system  is  critical  feature  of  the  system.  Our 
investigation  on  cancer,  diabetes,  autoimmune  disorders  and  cellular  mechanisms  all  revealed 
that  such  trade-offs  are  ubiquitous  and  inherent  properties  of  robust  evolvable  systems.  Carlson 
and  Doyle  pointed  out  that  robustness  may  be  a  conserved  value  and  has  robust  yet  fragile 
property  is  inherent  in  any  system.  This  is  very  important  feature  that  has  to  be  considered  in 
designing  engineered  system.  Studies  revealed  that  trade-off  does  not  only  exist  between 
robustness  and  fragility,  but  also  between  resource  demands  and  performance.  Thus,  robustness, 
fragility,  resource  demands,  and  performance  may  be  equated  in  a  set  of  equation.  This  implies 
the  system  designer  has  to  define  which  region  of  parameter  the  one  should  aim  at  in  designing 
the  system. 

The  system  that  has  high  level  of  resolution  requires  extensive  computing  power  to  process, 
and  could  be  fragile  against  certain  type  of  faults.  Information  fusion,  if  effectively  designed, 
may  compensate  for  fragility  of  such  high  performance  system  by  providing  alternative  sensory 
channel.  Of  course,  this  will  add  up  the  cost  and  may  sacrifice  overall  performance  of  the 
system  by  adding  weights  if  mobile  robot  is  assumed.  These  trade-offs  are  relevant  to 
information  fusion  and  integrated  autonomous  systems  that  has  to  operate  in  open,  dynamic, 
and  uncertain  environment. 

5.  Conclusion 

With  the  increasing  complexity  and  tendency  for  network-based  distributed  systems,  artificial 
systems  are  getting  closer  to  biological  systems.  Thus,  investigation  of  robustness,  information 
fusion,  behavioural  control,  and  evolution  of  biological  systems  shall  provide  us  with  rich 
insights  for  future  design  paradigm  of  distributed,  networked,  and  autonomous  engineering 
system.  In  the  distant  future,  the  author  believes  that  sufficiently  complex  engineering  systems 
share  substantial  architectural  similarities  with  biological  systems.  Lessen  shall  be  learned  now 
from  40  billion  years  of  design  challenges  through  the  evolution. 

The  work  performed  in  this  research  program  initially  investigated  possible  architecture  and 
approach  for  information  fusion  within  traditional  paradigms.  However,  it  became  apparent 
very  quickly  that  none  of  traditional  paradigm  actually  fits  to  overcome  inherent  difficulties 
facing  environments  that  are  dynamic  and  uncertain.  Thus,  aside  from  engineering  work  on 
sensory  and  robotics  systems  as  well  as  rescue  agent  simulation  system  that  are  not  mentioned 
here,  we  have  extended  our  scope  to  learn  from  biological  systems  that  have  actually  survived 
and  evolved  under  dynamics  and  uncertain  environment.  Some  of  insights  has  been  discussed 


in  this  executive  summary,  and  there  are  numerous  findings  that  will  be  documented  in  future 
as  results  of  this  research  program. 
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Chapter  1:  The  Architecture  for  Robust  Systems 


1.1  Robust  Architecture  for  Dynamical  Environments 


Abstract  -  The  fundamental  requirements  of  rescue  systems  are  robustness  against  perturbation  imposed  by 
disaster  and  practical  design  for  production  and  deployments.  Information  fusion  at  various  levels  needs  to  be 
designed  to  be  robust  and  scalable  to  size  and  types  of  disasters.  This  paper  discusses  basic  framework  of  the 
system  that  is  designed  to  cope  with  individual  and  home  securities,  but  can  be  converted  into  a  part  of  large 
system  in  case  of  large  scale  disasters.  It  follows  notion  of  robustness  that  are  found  in  many  engineering  and 
biological  systems,  and  explicitly  use  such  concept  for  basic  design. 


1  Introduction 

The  application  of  information  technology  to  rescue  human  life  in  disaster  is  one  of  the  major 
challenges  in  IT-related  research  with  high  impact  to  the  society.  Such  system  has  to  be  robust 
against  range  of  disasters,  but  also  has  to  be  cost  effective  and  easy  to  maintain  in  order  to  be 
practical. 

1.1  Scalability  is  an  economical  and  sociological  requirement 

When  considering  disaster  rescue  systems,  we  tend  to  assume  large  scale  disasters,  such  as  earth 
quakes  hit  Kobe,  Taiwan,  Turkey,  and  Iran.  However,  for  each  specific  area  to  suffer  disaster  of  this 
magnitude  is  rare.  In  some  cases,  it  happens  only  once  in  few  centuries. 

For  rescue  systems  to  be  practically  deployed  and  maintained,  as  well  as  being  economically 
justifiable,  it  has  to  be  able  to  cope  with  smaller  incidences  that  occur  in  daily  life. 

It  has  been  shown  that  size  of  earth  quake  and  its  frequency  has  specific  correlation  that  follows  the 
power  law.  The  power  law  distribution  is  observed  in  broad  range  of  phenomena.  It  would  be 
reasonable  to  assume  that  scale  and  frequency  of  general  disasters  and  crimes  also  follows  the 
power  law. 

This  implies  that  the  system  can  also  be  used  for  small  incidences  can  save  substantial  numbers  of 
people  aside  major  disasters.  In  addition,  because  the  system  will  be  maintained  in  daily  basis,  the 
system  is  more  like  to  be  functional  in  case  of  major  disaster. 

In  addition,  this  mitigates  one  of  the  major  issue  in  victim  search  for  the  large  scale  disaster  rescue 
that  is  how  to  get  sensors  near  the  victims,  when  location  of  victims  is  not  known  and  the 
environment  is  extremely  hostile.  Once  such  a  system  is  widely  accepted  and  used  in  the  daily  life, 
very  large  numbers  of  sensory  devices  are  readily  deployed  to  each  individual  and  to  each  house. 

The  design  of  the  system  has  to  be  scalable,  or  scale  free,  in  the  sense  the  system  deployed  for 
personal  use  can  be  also  an  integral  part  of  community  wide,  or  even  nation  wide  system. 

The  system  need  to  be  scalable  in  three  aspects; 


(1)  Problem  scalable  is  a  capability  of  the  system  to  cope  with  different  types  of  incidences, 

(2)  Size  scalable  is  a  system’s  ability  to  cope  with  incidences  with  different  magnitudes,  and 

(3)  Generation  scalable  that  ensures  the  system  to  be  continuously  up-graded  without  reinstalling 
the  entire  system. 

Particularly,  the  requirement  for  problem  scalability  where  the  system  to  cope  with  various 
incidences  including  attack  on  individual  working  on  the  street,  someone  breaking  into  the  house, 
fire,  a  major  earth  quake,  and  large  scale  terrorist  attacks.  Due  to  the  infrequencies  of  the  major 
disasters,  the  system  will  be  economically  feasible  and  sociologically  acceptable  only  when  it  can 
be  used  for  incidents  in  everyday  life.  This  enables  modules  to  be  manufactured  at  large  quantity,  so 
that  cost  can  be  drastically  reduced. 

Size  scalability  requirements  indicates  the  need  for  standard  architecture  and  protocols  that  enables 
such  modules  connected  to  form  network  of  systems,  regardless  of  manifacturers  and  original 
configurations. 

Generation  scalability  can  be  attained  also  by  assuming  stable  standard  architecture  and  protocols, 
so  that  modules  with  new  technologies  can  be  added  by  comply  with  the  standard  and  protocols,  as 
well  as  adding  consistent  additional  protocols  with  backward  compatibilities. 


1.2  Robustness  as  fundamental  architectural  principle 

Since  the  system  will  be  used  under  hostile  and  uncertain  environments,  robustness  shall  be  the 
major  concern  in  the  architectural  design  of  the  system.  Consideration  shall  be  given  at  all  levels  of 
system  to  enhance  its  robustness. 

Robustness  is  a  property  of  the  system  that  can  maintain  certain  properties  against  certain 
perturbations.  The  system  generally  exhibits  adaptation  against  changing  environments  and 
insensitivity  to  parametric  changes.  Such  properties  can  be  achieved  by  few  basic  mechanisms  that 
are;  (1)  feedback  control,  (2)  redundancy,  (3)  modularity,  and  (4)  structural  stability  [1,2]. 

For  example,  airplane  is  a  system  that  is  designed  to  achieve  stable  flight  against  taburance  of 
atomospheric  conditions  and  possible  damages  of  airplane  subsystems.  A  flight  control  system  can 
continue  to  maintain  its  ability  to  control  flaps  and  ladders  even  if  one  of  four  hyduralic  systems  is 
damaged.  This  is  possible  because  hyduralic  systems  are  built  redundant.  Automatic  flight  control 
enable  stable  flight  against  various  atomospheric  perturbations  due  to  its  feedback  control.  In 
addition,  it  is  composed  of  three  independent  flight  control  computers.  Three  computers  are 
developed  to  attain  identical  functions  (homogeneous  redundancy),  but  designed  as  differently  as 
possible  (heterogeneous  redundancy)  to  avoid  common  mode  failure. 

In  principle,  robustness  of  the  system  can  be  enhanced  by  designing  the  system,  composed  of 
modular  elements,  to  have  redundancy  in  various  aspects  and  impose  appropriate  feedback  control. 

In  the  context  of  this  research,  the  system  must  ensure  robust  detection  of  anomalies  and 
transmission  of  information  on  theater  of  disaster  under  various  types  of  disruptions. 


2  Universal  Security  Architecture 


Requirements  on  sealability  and  robustness  eonstraint  possible  design  spaee  for  arehiteeture  of 
general  seeurity  systems.  Naturally,  it  will  be  eonsists  of  well  defined  and  low  eost  multifunetional 
sensory  modules  with  reeonfigurable  eommunieation  eapability.  There  should  be  limited  types  of 
sueh  modules  forming  building  bloeks  of  the  system.  There  should  be  defined  protoeol  that  enables 
standard  eommand,  eommunieation,  and  eontrol  of  modules. 

Eaeh  module  shall  meet  defined  funetionalities,  but  designed  differently,  so  that  heterogeneous 
redundnaey  ean  be  maintained. 

Combination  of  modules  enables  various  funetions  to  eope  with  range  of  ineidenees  that  ensures 
sealability  regarding  its  problem  and  size. 

This  fundermental  arehiteeture  ean  be  illustrated  as  bow-tie  arehiteture,  that  is  elaimed  to  be  the 
fundermenal  arehiteeture  of  robust  systems  (Fig.  1). 
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Figrue  1;  The  bow-tie  arehiteeture  of  robust  systems 


USA:  Universal  Security  Architecture 


Figure  2  illustrate  an  example  of  how  sueh  systems  ean  be  implemented  for  individual  use. 
Numbers  of  sensor  modules  are  distributed  with  fixed  and  mobile  platform  that  are  aeeessed 
through  home  seeurity  server  or  direetly  through  mobile  eommunieation  networks.  Faeh  module 
ean  be  funetion  as  probes  for  sensing  vietims  in  ease  of  large  seale  disaster. 

3  Module  Design 

3.1  Module  Functionalities 

The  basie  element  of  the  system  is  a  multi-funetional  sensory  module  that  is  small  and  versatile. 
The  module  is  designed  to  be  robust  in  sensing  its  own  environment  and  eommunieating  with  other 
modules.  While  there  are  several  types  of  basie  modules,  an  example  of  basie  feature  is  as  follows: 


(1)  Sensors:  CCD  or  CMOS  image  sensors,  Infra-red,  taetile,  mierophone 

(2)  GPS  and  Gyro 

(3)  Speakers 

(4)  Lights 

(5)  Wired,  Wireless,  and  IR  communieation  ehannels 

(6)  Battery  opertion  capability  (includnig  additional  solar  panels) 

Sensors  can  detect  situations  around  the  module.  GPS  and  gyro  can  identify  location  and  posture  of 
the  module.  Speakers  can  be  used  to  (1)  call  victims  asking  for  their  reply,  (2)  communicate  with 
victims  once  found,  (3)  communicate  with  other  modules  using  a  low-speed  communication 
protocol,  and  (4)  to  notify  other  modules  and  human  rescue  staff  where  the  module  located.  Lights 
are  used  to  (1)  to  probe  dark  confined  envrionment  by  imaging  devices,  (2)  communicate  with  other 
modules,  and  (3)  indicate  location  of  the  module  to  other  modules  and  human  rescue  staff.  Various 
broadband  and  narrow  band  communication  channels  are  mostly  used  for  daily  life  or  home  and 
indiviudal  security  purposes,  but  can  be  used  in  large  disasters  if  such  infrastructures  are  still 
available.  Since  we  cannot  assume  electricity  in  major  disaster  stiuation,  modules  should  be 
operational  with  battery. 

Such  versatile  module  can  be  placed  in  house  or  mounted  on  mobile  platforms,  such  as  robots  or 
automobile. 


3,2  Robsutness 

Robust  sensing  is  attained  by;  (1)  the  use  of  multiple  sensor  units,  (2)  the  use  of  multiple  modalities 
of  sensing,  and  (3)  adaptive  control  of  sensory  units. 

3.2,1  Redundancy 

The  module  is  equipped  with  sensors  for  different  type  of  signals,  such  as  CCD  or  C-MOS  image 
devices  for  normal  visual  spectrum,  infrared  sensors,  microphone,  tactile  sensors,  etc.  This  ensures 
robustness  through  heterogeneous  redundancy  in  sensing  environment  under  possible  perturbations, 
such  as  undesired  lighting  that  disable  normal  vision  sensors. 

Each  sensory  module  is  expected  to  have  multiple  units  within  a  module,  so  that  damage  on  one  of 
the  sensors  do  not  directly  leads  to  total  dysfunction  of  the  system.  At  the  same  time,  the  use  of 
multiple  modules  for  proximal  area  would  add  robustness  as  a  whole  system. 


We  have  designed  and  developed  small  sensory  modules  that  has  multiple  imaging  devices  to  cover 
broad  range  of  visual  field  (Figure  3). 


Figure  3:  Prototype  of  sensory  module 


3,2,2  Adaptation 

Adaptation  against  environmental  stimuli  can  be  accomplished  by  (1)  actively  moving  sensory 
device,  and  (2)  changing  the  amplihcation  gain. 

Identification  of  the  direction  of  sound  source  is  one  of  the  features  that  is  highly  useful  in  locating 
victim  location.  However,  due  to  physical  property  of  sound,  directional  resolution  is  maximized 
when  microphone  is  aligned  tangential  to  the  sound  source.  Thus,  active  audition  system  has  been 
developed  that  can  react  to  possible  sound  source  direction,  and  turn  microphone  units  to  maximize 
directional  resolution  [3-5].  This  can  be  considered  as  extension  of  animated  vision  and  active 
vision  concept  to  audition  [6]. 


Bacteria  exhibits  high  level  of  robustness  in  chemotaxis  [7].  With  gradient  chemoattractant 
environment,  bacteria  swim  toward  high  concentration  area,  and  such  a  behavior  is  maintained  for 
broad  range  of  concentration  level.  It  is  attained  by  the  use  of  integral  feedback  involving  the 
receptor  complex,  frageha  motors,  and  intermediate  enzymes  (Figure  4)  [8].  This  feedback  system 
react  to  changes  in  the  concentration  of  chemoattractant,  instead  of  absolute  concentration  level. 
When  concentration  level  of  chemoattractant  changes,  receptor  activity  rate  changes  that  triggers 
change  in  frageha  motor  activity.  However,  feedback  control  acts  to  bring  activity  level  of  this 
module  to  the  steady  state.  Because  of  this,  bacteria  can  react  to  further  stimuli.  Such  adaptation 
against  external  stimuli  is  ubiquitously  observed  in  bioloigcal  signal  transduction. 


Figure  4:  Feedback  control  in  bacteria  chemo taxis 


Similar  principle  can  be  applied  to  sensory  systems.  First,  dynamics  range  of  device  can  be 
augmented  by  implmenting  a  feedback  control  to  adjust  signal  intensity  that  reach  sensing  device. 
Auto-iris  is  one  of  such  feedback  systems  that  are  implemented  on  engineering  systems. 
Microphone  dynamics  range  can  be  augmented  by  creating  physical  devices  that  can  adjust  sound 
level  reaching  the  microphone.  Second,  gain  adjustment  of  the  amplifier  connected  to  the  sensing 
device  provides  stable  signal  coping  with  changes  in  the  incoming  signals. 

In  addition  to  such  signal  level  adaptation,  anomaly  detection  requires  distinction  between  normal 
envrionment  with  certain  fluctuation  and  anomaly.  Fro  example,  if  sun  lights  into  the  room 
gradually  changes  over  time,  it  is  normal  due  to  movement  of  the  sun.  However,  if  it  changed 
drastically  within  a  few  seconds,  it  may  due  to  someone  manipulated  room  environment  which  is  a 
potential  anomaly.  Integral  feedback  as  seen  in  bacteria  can  cope  with  such  changes,  but  additional 
mechanisms  need  to  be  implemented  to  avoid  false  positive  that  such  manipulaiton  is  done  by  the 
family  members.  Some  securty  key  need  to  be  implemented  that  supercedes  automatic  anomaly 
detection. 

3.3  Information  Fusion 

Automatic  detection  of  anomalies  and  sending  the  message,  as  well  as  recording  the  data,  is  one  of 
the  major  functions  that  are  expected  in  daily  use  of  such  devices. 

Aan  extreme  AI  solution  is  that  each  module  to  have  intelligence  program  that  can  distingush 
anomalies  and  normal  perturbations  and  send  warning  only  when  anomaly  take  place.  However, 
this  is  not  practical  as  false  positive  can  not  be  eliminated  if  the  system  is  tuned  to  detect  all 
anomalies. 

Practical  solution  would  be  to  combine  the  system  with  physical  and  electric  security  keys,  as  well 
as  pre-designed  sensor  neworks.  Security  keys  can  override  anomaly  detection  function  of  the 
module,  so  that  actions  taken  with  securty  key  activated  will  not  be  detected  as  anomaly.  At  the 
same  time,  convnetional  home  security  systems  places  infrared  and  open-close  switches  to  doors 
and  windows,  so  that  movement  of  objects,  doors,  and  windonws  can  be  detected.  Users  can  define 
the  mode  of  operation,  so  that  the  system  know  which  sensors  has  to  be  activated.  Such 
conventional,  and  somewhat  low-tech  solution  is  quite  effective. 


Then,  where  is  need  for  intelligenee  systems?  Although  most  low  teeh  systems  ean  deteet  anomalies 
if  it  is  well  plaeed,  it  ean  not  provide  images  of  detailed  situaiton  and  send  sueh  information 
effeetively  to  users.  Usrs  are  most  likely  to  be  in  the  situation  aeeess  to  the  visual  information  is 
restrieted  or  ean  use  only  limited  size  of  sereen  sueh  as  mobile  phone.  In  sueh  eases,  looking  at 
images  of  numbers  of  deviees  are  time  eonsuming,  and  may  not  be  able  to  find  anomalies,  even  if  it 
is  reported  and  eaptured  in  one  of  the  image  sensors. 

Intelligent  information  fusion  would  assists  sueh  situation  by  providing  images  that  are  seored 
highly  suspieious  to  the  user  when  the  user  first  logged  in  to  the  system. 

Also,  it  may  able  to  traeking  the  movement  of  suspieious  objeets  in  the  house. 

Information  fusion  plays  a  eritieal  role.  Vision  is  often  oeeruded  and  may  not  be  able  to  eover  an 
entire  area.  Autditory  systems,  on  the  other  hand,  may  be  able  to  deteet  sounds  even  if  objeets  are 
oeeruded  visually.  With  eertain  levels  of  direetion  identifieation  eapability,  it  may  be  able  to  loeated 
where  the  suspeets  may  be  hiding  in  the  house. 

In  addition,  using  both  visual  and  auditory  information  is  shown  to  reduee  false  positives  [9]. 

The  other  possiblity  is  the  linkage  with  early  warning  system  for  earth  quake.  Using  the  time  of 
earth  quake  vibration  to  propagate,  the  system  has  been  already  implemented  in  Japan  that  ean  send 
warning  sueh  as  ”  an  earth  quake  with  intensity  5  to  arrive  8  seeonds  from  now”.  If  sueh  signal  ean 
be  transmitted  in  maehine  readable  form,  modules  ean  be  immidiately  aetivated  to  reeord  situation 
before  the  arrival  of  the  earth  quake,  and  reeord  how  damages  are  inflieted.  Sueh  information  would 
be  enormous  help  to  reseue  teams,  as  they  ean  understand  who  may  be  around  and  how  eollapse 
took  plaee. 

4  Robust  Communication  among  Modules 

Modules  need  to  self-organize  their  network,  so  that  information  of  the  environment  of  eaeh  module 
ean  be  transmitted  to  outside  of  the  area,  so  that  rescue  teams  can  understand  the  situation,  and  act 
accordingly. 

4,1  Redundancy 

Each  module  have  different  communication  modalities,  including  wired  eithernet,  wireless  LAN  or 
mobile  phone,  infra-red,  sound,  and  light.  Basically,  each  channels  use  same  protocol,  but  modified 
according  to  band  width  of  each  modality.  Communication  channels  are  made  redundant  in  a  sense 
all  these  channels  can  communicate  to  other  modules  in  normal  condition.  At  the  same  time,  it  is 
heterogeneous  because  different  mode  of  communication  are  selected  to  add  robustness  against 
unpredictable  situations. 

After  the  initial  shock,  surviving  modules  start  to  establish  network  of  communication.  Ech 
channels  are  tried  to  estblish  connections.  When  wired  ether  net  is  tried  but  failed,  then  wireless 
LAN  or  mobile  phone  channel  is  tried.  When  it  failed,  use  light  or  IR  channel  to  communicate  with 
nearest  modules. 


Mob i I e 


Mob i  I  e 


Figure  5:  Emergent  network  based  on  survived  modules 

Figure  5  illustrates  possible  eonfiguration  of  the  network.  Module  1  and  2  are  visually  partitioned, 
so  that  only  sound  ean  be  used  to  eommunicate  between  these  modules.  Module  2  ean  send  signals 
to  module  3  usnig  lights,  but  module  3  can  not  send  sufficient  lights  to  module  2,  so  that  sound 
channel  is  used  instead.  Module  3  can  communicate  with  module  5  thrhough  mobile 
communication  channel.  It  is  most  likely  that  this  module  to  be  the  gatewya  to  larger  networks. 
Module  3  can  send  signal  to  module  4  via  light,  but  module  4  can  not  communicate  with  any  other 
modules. 

The  communication  program  on  the  module  has  to  be  flexible  to  allow  such  heterogeneous  network 
with  different  physical  properties  and  band-width. 

Ideally,  triangulation  shall  be  done  using  sensros  and  GPS  and  gyro,  so  that  physical  configuration 
can  be  estimated. 


4.2  Adaptation 

The  problem  is  how  to  spontaneously  organize  the  network  topology  with  heterogeneous  modalities 
and  often  interrupting  channels. 

An  interesting  observation  has  been  shown  that  in  the  small  world  network,  any  two  node  can  be 
communicated  within  small  numbers  of  hops,  such  as  6  or  7  hops.  This  can  be  attained  by  growing 
the  network  accodring  to  preferential  attachment  principle.  The  priferential  attachment  means  that 
node  with  highly  connected  with  other  nodes  are  more  likely  to  have  extra  connections,  so  that  it 
spontaneously  creates  hubs  in  the  network.  The  advantage  of  such  network  is  that  it  is  robust 
against  random  failure  of  nodes.  In  this  context,  robustness  is  measured  by  the  stability  of  the 
average  number  of  hops  between  arbitrary  chosen  two  nodes  in  the  network. 

This  is  desirable  property  for  nework  connecting  sensory  modules  for  victim  search,  because 
communication  network  topologies  can  not  be  determined  in  advance  and  an  be  interupted  by 
failure  of  any  of  nodes  in  the  network. 


5.  Active  network  reconstruction  by  mobile  modules 


After  the  reeonstruetion  of  networks  by  stationary  modules,  the  mobile  systems  may  be  aetivated  to 
eover  the  areas  that  are  not  eovered  by  the  stationary  modules.  This  requires  eertain  level  of 
identifieation  ean  be  made  on  the  loeation  of  eaeh  modules  and  networks.  Then,  the  task  is  to 
deploy  mobile  modules  where  sueh  networks  are  not  established,  or  the  grain  size  to  eover  the  area 
is  not  suffieient. 


Figure  6:  Feedbaek  loop  of  information  fusion  and  notions 

Deoision  on  how  to  deploy  mobile  modules  may  be  under  the  olosed  loop  of  low-level  sensory 
fusion,  high-level  information  fusion,  and  deoisions  followed  (Figure  6)  [10]. 


6  Adaptive  airborne  deployments  for  spatial  information  fusion 

Given  the  spatially  distributed  and  soattered  nature  of  initial  networks,  theater  seale  deployments  of 
airborne  modules  to  effieiently  fill  the  gap  would  be  effeetive. 

The  author  has  previously  proposed  airborne  deployments  of  sensor  modules  where  no  sensors  are 
available  on  the  ground  [10].  The  scenario  in  this  paper  is  that  there  are  fragmented  networks  of 
modules  are  available  on  the  ground,  but  scattered  and  global  triangulation  is  not  completed. 

In  this  case,  adaptive  formation  of  modules  are  modified  to  focus  on  the  area  ground  based  modules 
are  not  clustered,  and  small  triangulated  modules  are  diverted  into  the  fragmented  network  to 
identify  spatial  location  as  well  as  to  ensure  its  connection  to  global  network  to  be  establish  after  the 
arrival  of  the  airborne  modules  (Figure  7). 


Aerial  Distributed  Inforntion  Fusion 


Phase- 1 


Mid-air  Deploynent 


Phase-il:  Configuration  Identification  (triangulation) 


Phase-lll:  Synthetic  Terrain  Mapping  and  Recognition  of  Fragmented  Ptetvorks 


Phase-V:  Construction  of  Coatiined  Networks  and  Ground-Based  Information  Collection 


Figure  7:  Airborne  deployment  of  modules  for  aerial  terrain  mapping  and  network  reconstruction 

7  Conclusions 

In  this  paper,  basic  design  strategies  for  disaster  rescue  systems  have  been  discussed.  The 
fundamental  principle  behind  the  design  is  scalability  and  robustness.  For  such  a  system  to  be 
practical  and  acceptable,  it  must  be  able  to  cope  with  incidents  in  the  daily  life,  in  addition  to  rare 
events  of  large  scale  disasters.  This  is  the  only  way  to  make  systems  affordable,  deployable,  and 
maintainable.  Redundancy  in  sensory  systems  and  feedback  control  shall  provide  robust  perception 
and  detection  of  anomalies.  Emergent  network  of  communication  enable  the  modules  to 
communicate  and  transmit  information  to  rescue  teams,  as  well  as  sending  messages  of  rescue 
teams  to  victims  in  debris.  Active  probing  using  ground  based  mobile  modules  as  well  as  theater 
scale  airborne  deployment  shall  provide  global  network  by  connecting  fragmented  networks  and 
enable  us  to  understand  pictures  of  the  terrain  globally. 

It  is  important  at  this  moment,  overall  framework  of  the  system  to  be  investigated  and  defined. 
Details  of  each  technical  aspect  shall  follow  once  overall  architecture  is  defined  as  attempted  in  this 
paper. 


1.2  Challenges  in  Robust  Situation  Recognition  through 
Information  Fusion  for  Mission  Criticial  Multi- Agent  Systems 


The  goal  of  this  paper  is  to  highlights  one  of  emergent  seientifie  issues  in  RoboCup  task  domains 
that  has  broader  applieations  even  outside  of  the  RoboCup  task  domains.  This  paper  partieularly 
foeuses  on  robust  reeognition  through  information  fusions  issue  among  numbers  of  others  issues 
that  are  equally  important.  The  robust  reeognition  through  information  fusion  is  seleeted  beeause  it 
is  one  of  the  most  universal  issues  in  AI  and  roboties,  and  partieularly  interesting  for  domains  sueh 
as  soeeer  and  reseue  that  has  high  degree  of  dynamies  and  uneertainty,  as  well  as  being  resouree 
bounded.  The  author  wish  to  provide  a  coneeptual  framework  on  robust  pereeption  from  single 
agent  to  multi-agent  teams. 


Robustness  and  Information  Fusion 

The  RoboCup  projeet  is  an  interesting  researeh  platform  beeause  it  has  multiple  domains  that  have 
both  differenee  and  similarities  in  basie  domain  features.  Soeeer  is  a  dynamie  game  with  real-time 
eollaboration  within  teammate,  but  adversarial  against  opponents.  Major  uncertainties  are  generated 
by  (1)  opponent  strategies,  (2)  uncertain  and  stochastic  nature  of  physics  on  ball  movements  and 
other  factors  that  affect  course  of  each  action,  and  (3)  uncertainty  of  real-time  behaviors  of  each 
player.  On  the  contrary,  rescue  domain  is  a  dynamic  and  mission  critical  domain  with  high  degree 
of  uncertainty  and  partial  information  under  hostile  terrain  that  are  totally  different  in  each  case. 
Major  uncertainties  are  generated  by  (1)  unknown  terrain  and  victim  information,  (2)  uncertain  and 
stochastic  nature  of  incoming  information,  success  of  each  operations,  numbers  of  external 
perturbations,  and  other  social  and  political  factors,  and  (3)  limited  information  on  individual 
victims  and  their  situations. 

It  is  interesting  to  note  that  despite  differences  in  the  task  domain,  there  are  substantial 
commonalities  in  structures  of  uncertainty.  There  are  uncertainty  and  limited  information  at  the 
macroscopic  level  at  the  level  of  entire  terrain  or  theater  of  operation  and  at  the  microscopic  level 
which  is  the  scale  of  individual  players  or  victims.  In  addition,  there  are  issues  of  unknown  and 
unpredictable  perturbations  throughout  the  operation. 

In  order  to  best  accomplish  the  task,  a  team  (or  teams)  of  agents,  either  robotics  or  informational 
agents,  need  to  be  robust  in  perceptions  and  actions,  as  well  as  their  team  behaviors.  It  should  be 
well  coordinated  to  reconstruct  a  model  of  the  terrain  and  other  agents  in  the  scene  against  various 
noise,  uncertainty  and  perturbations,  so  that  effective  actions  can  be  taken.  A  set  of  actions  need  to 
be  robust  so  that  failure  of  one  or  more  of  such  actions  do  not  leads  to  catastrophic  failure  of  the 
overall  mission. 

Robustness  of  the  system  is  generally  exhibited  as  capability  of  the  system  to  (1)  adapt  to 
environmental  fluctuation,  (2)  insensitivity  against  fluctuations  in  system's  internal  parameters,  and 
(3)  graceful  degradation  of  performance,  as  opposed  to  catastrophic  failure  of  the  system. 

This  applies  from  sensory  level  to  higher  multi-agent  team  level.  A  brief  example  at  the  team  level 
shall  make  clear  what  does  robustness  means.  For  the  soccer  team,  this  means  that  the  team  should 
be  able  to  cope  with  differences  in  strategy  and  physical  performance  of  opponent  team,  not  being 
seriously  affected  by  changes  in  fatigue  and  other  factors  of  players,  and  removal  of  one  or  more 


players  does  not  results  in  complete  loss  of  its  team  performance.  For  the  rescue  team,  it  means  that 
the  team  can  cope  with  various  different  disaster  scenario  and  dynamical  changes  in  the  situation, 
ability  of  cope  with  unexpected  fatigue,  damage,  and  resource  shortage,  and  capability  to  carry  out 
missions  even  if  some  of  its  rescue  teams  have  to  be  withdrawn  from  the  scene.  For  the  rescue  team 
that  has  to  cope  with  extremely  hostile  environment,  robustness  is  one  of  the  essential  features  of 
the  system. 

Robustness  of  the  system  is  usually  attained  by  (1)  the  use  of  proper  control,  such  as  negative 
feedback,  (2)  redundancy,  or  overlapping  functions  of  multiple  subsystems,  (3)  modular  design,  and 
(4)  structural  stability. 

Extensive  research  has  been  made  on  properties  of  robustness  in  biological  systems,  and  these  traits 
were  shown  to  be  universal  using  numbers  of  examples,  including  bacteria  chemotaxis,  robustness 
of  gene  transcription  against  point  mutations  and  noises,  stable  body  segment  formation,  cell  cycle 
and  circadian  period,  etc. 

Bacteria,  for  example,  swim  toward  chemoattractants  by  sensing  graduent  of  concentration.  This 
capability  is  maintained  regardless  of  concentration  level,  steepness  of  the  gradient,  and  keep  track 
of  graduent  changes  consistently.  Integral  feedback  has  been  identified  as  a  key  intrinsic  mechanism 
in  which  bacterial  behaviors  are  controlled  by  activation  of  receptor  complex,  but  deactivated  by  a 
negative  feedback  loop  with  integral  components.  This  feedback  control  enables  behavior  of 
bacteria  dependent  on  the  level  of  concentration  changes  took  place,  but  independent  of  absolute 
concentration  level  of  chemical  in  environment.  Similar  mechanisms  are  observed  widely  amoung 
different  speceies.  Feedback  control  is  only  one  of  several  mechanisms  behind  biological  robustness. 

On  the  contrary,  artificial  systems  tend  to  be  less  robust,  and  reply  on  rigid  build-in  design  that  may 
easily  fail  under  fluctuations.  How  to  build  robust  systems  from  sensory-level  to  strategy-level  in  a 
consistent  manner  is  one  of  the  major  issues  in  RoboCup  research. 

In  the  rest  of  the  paper,  possible  research  directions  for  robust  systems  particularly  focusing  on 
information  fusion  aspects  are  discussed.  Information  fusion  is  raised  here  because  it  is  relatively 
universal  in  different  domains,  and  critical  for  strategic  actions  to  follow.  Issues  of  robust  strategic 
decisions  and  executions  will  be  the  other  robustness  issues  in  multi-agent  teams,  but  will  not  be 
discussed  due  to  space  limitations. 

Dimensions  of  information  fusion  for  robust  systems 

Information  fusion  for  robust  systems  has  to  be  considered  for  multiple  aspects: 

*  Abstraction:  An  interactive  processing  of  different  abstraction  levels,  such  as  interactive 

processing  of  low-level  sensory  information  and  high-level  recognition  and  strategy,  enhances 
robustness  by  providing  interlocking  feedback  of  information  thereby  hypotheses  may 
converge  to  more  plausible  one. 

*  Multiple  Sensory  Channels:  Integration  of  multiple  modal  perception  channels  can  contribute  to 

improve  robust  perception  by  complementing  missing  information  by  perception  channels  with 
different  characteristics. 

*  Perception-Action  Loop:  Active  involvement  of  probing  actions  into  perception,  that  is  an 

integration  of  perception-action  loop  to  enhance  recognition  by  actively  manipulating  the 
environment  so  that  ambiguity  can  be  resolved. 


*  Spatio-Temporal  Distribution;  Integration  of  spatio-temporally  distributed  sensory  information, 
as  well  as  absteacted  symbolie  information  is  essential  to  create  overall  picture  of  the  situation, 
thereby  robust  adaptation  to  the  situation  can  be  done  with  overall  re-evaluation  of  the  situation. 


Information  fusion  in  these  aspects  contributes  robust  perception  of  theater  of  operation  through  one 
or  more  of  four  mechanisms  of  robustness. 


Interaction  of  low-level  and  high-level  perception  and  actions 

Given  the  multi-scale  nature  of  the  domain  that  requires  identification  of  situation  at  both 
macroscopic  and  microscopic  levels,  distributed  and  coordinated  information  fusion  need  to  be 
performed  that  are  ultimately  combined  to  create  a  coherent  model.  Information  fusion  at  the 
macroscopic  level  is  called  the  high  level  information  fusion  (HiLIF)  and  that  of  the  microscopic 
level  is  called  sensory  level  information  fusion  (SLIP). 

In  the  sequential  model  of  AI,  a  famous  Sense  Model  Plan  Act  cycle  (SMPA  cycle)  has  been  used. 
This  paradigm  has  been  criticized  as  not  being  able  to  respond  to  environment  in  real-time,  and  an 
alternative  approach  named  "behavioral-based  approach"  has  been  proposed. 

While  the  behavior-based  AI  demonstrated  effectively  how  simple  robotics  systems  and  virtual 
agents  can  behave  in  the  real  world  without  creating  an  internal  model,  it  never  scaled  to  perform 
complex  tasks.  In  both  soccer  and  disaster  rescue,  coupling  of  hierarchy  of  sensing  and  actions  from 
low-level  sensory  units  to  high-level  strategy  generators  is  essential.  Reactive  control  of  leg 
movement  to  kick  a  ball  must  be  coordinated  with  strategic  choice  of  attacking  patterns  and  a  pass 
trajectory  to  enable  such  a  strategy.  Similarly,  local  actions  and  decision  for  the  recovery  of  some 
critical  life-lines  has  to  be  carefully  coordinated  with  overall  strategy. 

In  this  context,  low-level  perception  and  action  module  is  not  merely  behavior-based  module.  It 
must  be  able  to  recognize  local  situation  that  can  be  aggrigated  at  a  higher  level.  SLIP  should  have  a 
certain  level  of  internal  model. 

While  these  examples  are  how  overall  strategy  may  constrain  local  actions,  there  are  cases  local 
actions  influence  overall  strategy.  Basically,  it  is  interaction  of  bottom-up  and  to-down  processes  in 
which  the  architecture  enabling  it  has  been  long  standing  theme  in  multi-agent  systems.  While  this 
is  well  recognized  issue,  the  author  would  not  make  further  discussions  than  stating  that  RoboCup 
task  domains,  particularly  humanoid  league  and  rescue,  are  one  of  ideal  platform  for  seriously 
tacking  this  problem. 

This  aspect  of  integration  essentially  exploits  feedback  control  of  the  system  for  adaptation  at 
certain  abstract  levels.  Low-level  perceptions  and  local  decisions  are  aggrigated  to  higher-level  that 
feedback  differences  between  desired  local  situations  and  actions  and  actual  situation  and  actions  to 
reduce  such  descrepancies. 


Information  fusion  of  multiple  sensory  channels 

Navigation,  localization,  and  object  identification  of  robotic  agents  tends  to  rely  on  visual 
information.  While  vision  provides  rich  information,  it  is  limited  in  several  aspects.  First,  visual 
perception  can  be  occluded  by  obstacles,  including  dust  and  smokes.  Second,  it  has  to  have  certain 
quality  of  light  sources.  These  features  impose  serious  limitations  of  visual  perception  for  disaster 
rescue  situation,  because  disaster  scenes  are  generally  highly  unstructured,  dusty,  and  may  have 


serious  smokes  from  fire.  Lights  are  often  not  available  in  confined  environment  where  victims  may 
be  embedded. 

Auditory  perception,  on  the  other  hands,  has  different  characteristics.  It  can  transmit  over  obstacles, 
do  not  require  light  sources.  In  fact,  various  noises  victims  may  make  is  a  critically  important 
signals  for  a  victim  location  identification.  Other  sensory  channels,  such  as  odorant,  C02,  and 
vibrations  have  different  characteristics  that  complement  each  other. 

Information  of  multiple  modalities  of  perceptions  may  provide  high  level  of  robustness  in 
perceiving  the  environment.  Some  of  early  efforts  have  been  done  using  integration  of  auditory  and 
visual  perception. 

Vision  system  often  generates  false  positive  identification  of  objects  that  is  supposed  to  recognize 
due  to  similarity  of  color  and  shape  in  irrelevant  objects.  When  the  object  is  creating  certain 
auditory  signals,  the  use  of  auditory  information  to  track  the  sound  stream  can  effectively  eliminates 
false  positives.  By  the  same  token,  objects  could  be  occluded  by  obstacles  so  that  vision  system  lost 
tracking,  which  can  be  compensated  by  keep  tracking  auditory  signals  if  the  object  is  making  some 
sound  streams.  Early  experiments  indicate  that  well  designed  integration  of  multiple  modal 
perception  channels  based  on  multiple  sensory  steam  integration  is  effective  for  robust  perception. 
Now,  the  research  topic  shall  be  to  create  basic  principle  of  robust  object  identification,  tracking, 
and  scene  understanding  by  using  multiple  perception  channels,  most  likely  by  defining  a 
dynamical  defined  invariance  that  corresponds  to  each  object  as  its  signature. This  approach  is 
expected  to  attain  robustness  by  exploring  redundancy,  or  overlapping  functions,  of  perception 
channels,  so  that  degradation  or  failure  of  one  perception  channel  is  well  compensated  by  other 
perception  channels  with  overlapping  functions. 


Integrating  actions  to  perception 

It  is  important  that  the  concept  of  active  perception  to  be  integrated  into  the  system,  so  that 
ambiguities  of  information  can  be  resolved  by  actively  directing  sensory  devices  and  information 
gathering  agents.  Early  research  of  this  direction  has  been  proposed  as  active  vision  and  animated 
vision  \cite {Ballard},  but  it  has  to  be  extended  to  include  not  only  vision  and  other  perception 
channels,  but  also  to  more  high  level  information  collections.  Conceptually,  this  is  feedback  control 
to  minimize  unknown,  or  ambiguous  part  of  scene  to  zero. 

In  the  resource  constraint  situation  such  as  in  disaster  rescue,  the  concept  of  cost  of  active  probing 
has  to  be  introduced.  Any  action  to  obtain  information  is  associated  with  cost,  the  use  of  resources, 
including  time.  Decision  has  to  be  made  on  whether  actively  probe  new  information  or  to  proceed 
with  ambiguities. 

Suppose  that  a  ball  that  is  rolling  from  right  to  left  is  occluded  by  an  opponent  player,  decision  has 
to  be  made  to  use  predictions  of  ball  trajectory  or  actively  probe  the  ball  position.  While  actively 
proving  the  ball  position  may  resolve  ambiguity  of  information,  it  may  loose  time  window  to 
intercept  the  ball. 

By  the  same  token,  in  the  disaster  scenario,  spreading  of  fire  or  cascading  collapse  of  buildings  may 
not  be  fully  monitored  by  available  sensors  or  information  agents.  A  tactical  decision  has  to  be 
made  to  dispatch  a  unit  to  counter  such  incidence  by  predicting  possible  front  of  chain  reactions  or 
to  mobilize  information  gathering  agents  to  make  sure  the  status  of  incidents.  The  cost  of 
information  gathering  is  the  use  of  additional  agents  and  time  to  wait  until  the  situation  to  be 
disambiguated.  On  the  contrary,  a  quick  deployment  of  counteraction  units  runs  a  risk  that  the 
prediction  is  false  and  deployments  are  deemed  ineffective.  Always,  there  is  a  trade-off  between 
cost  of  knowing  and  risk  of  not  knowing.  One  of  the  research  topics  may  be  to  find  out  principles  of 
decision  on  cost  of  knowing  versus  risk  of  not  knowing. 


Spatio-temporal  integration 


Heuristics  and  knowledge-based  estimation 

Integration  of  spatio-temporal  information  is  essential,  partieularly  for  disaster  reseue  operations. 
Theater  of  operation  is  widely  distributed,  and  information  is  eolleeted  only  at  limited  rate  from 
limited  loeations.  Basieally  it  is  a  problem  of  making  the  best  estimate  of  the  situation  by  sparse 
sampling  of  eomplex  terrain  in  4-D  (XYZ+T)  spaee.  Certain  heuristies,  sueh  as  eontinuity  of  the 
unfolding  events,  and  a  priori  on  strueture  of  urban  infrastrueture  are  expeeted  to  be  highly  useful  to 
eonstrain  possible  hypotheses.  This  applies  to  both  low-level  and  high-level  information  fusion,  but 
particularly  useful  for  high-level  information  fusion  where  "fog  of  war"  has  to  be  resolved  as  soon 
as  possible  with  limited  resources.  Advantage  of  this  approach  is  that  you  can  make  reasoned 
estimate  of  the  situation  even  for  the  area  that  cannot  be  directly  measured.  The  drawback  is  that  it 
requires  substantial  knowledge  of  the  urban  structures  in  usable  form  that  are  generally  not 
available. 

At  the  same  time,  how  to  increasing  sampling  points  is  the  other  big  issue.  One  of  the  best  ways  is 
to  make  sure  sensory  systems  are  ubiquitously  present  in  disaster  scene.  This  can  be  achieve  only 
by  creating  multi-functional  systems  that  are  useful  in  daily  life,  but  can  act  as  sensory  units  in 
emergency.  Traffic  signals  and  various  monitoring  cameras  are  possible  resources  that  can  cover 
public  space.  Home  security  systems,  home  entertainment  robots,  and  a  series  of  home  electronic 
products  are  ideal  for  covering  situations  in  each  household.  However,  there  are  issues  of  securing 
telecommunication  with  such  devices,  as  well  as  protection  of  privacy  that  are  critical,  but  are 
outside  of  AI  and  robotics  issues. 


Adaptive  airborne  spatial  information  fusion 

One  of  possible  approach  to  solve  this  problem  is  to  develop  a  small  disposable  sensory  unit  and 
deploy  them  in  large  numbers.  Figure  2  illustrates  one  example  of  such  systems  which  could  be 
deployed  airborne.  The  goal  is  quickly  understand  target  terrain/city  situation  for  the  purpose  of 
disaster  rescue. 


*  Phase-I;  Large  number  of  small  IF  (information  fusion)  devices  will  be  deployed  mid-air 

*  Phase-II;  Speed  break  is  activated  and  speed  is  stabilized.  Then  each  unit  has  some  visual  beacon 

or  visual  ID  so  that  relation  location  of  units  can  be  determined  by  triangulation. 

*  Phase-III;  First  aerial  photo/video  will  be  taken  using  multiple  camera  unit,  but  mostly  using 

cameras  that  are  facing  terrains  below,  and  send  back  to  airborne  server  or  other  servers.  Focus 
of  attention  is  determined  and  rough  model  will  be  created.  If  processing  power  is  sufficient, 
this  can  be  done  in  each  unit.  Infra-red  sensors  or  other  sensors  can  be  used  in  addition  to 
CMOS  imager  to  identify  specific  signature  on  terrain. 

*  Phase-IV:  Each  unit  deploys  small  and  simple  flaps  to  control  trajectory,  so  that  visual  servo  will 

be  made  effective  to  regroup  IF  units,  so  that  areas  that  are  more  significant  will  be  assigned 
with  dense  IF  units.  Photo/video  analysis  and  reconstruction  continues.  In  low  altitude,  all  360 
angle  camera  as  well  as  microphone  and  other  sensors  are  activated  to  capture  maximum 
information  on  the  ground.  If  possible,  frame  rate  will  be  increase  dramatically. 


*  Phase-V:  For  those  units  that  safely  reached  the  ground  and  survived  impact,  360  degree  vision 
and  microphone  systems,  as  well  as  other  sensors  will  be  activated  to  identify  objects  and 
terrain  structure. 

This  approach  integrates  self-organization  of  agents  and  sensory  feedback  on-the-fly,  and  can  be 
enormously  useful  for  rapid  deployment  at  disaster  site,  as  well  as  being  an  excellent  test  of  modern 
AI  techniques. 

Conclusion 

This  paper  addressed  several  issues  in  robust  systems  for  RoboCup  domains.  Several  aspects  of 
robustness  have  been  identified,  and  four  aspects  of  information  fusion  have  been  discussed  briefly. 
Both  soccer  and  rescue  provides  an  excellent  platform  for  such  research,  and  several  research  issues 
have  been  raised.  Integration  of  three  dimensions  (abstraction  levels,  perception  modality,  and 
perception-action  loops)  of  information  fusion  poses  particularly  interesting  problems  that 
community  shall  tackle.  Spatio-temporal  integration  applies  to  both  soccer  and  rescue,  but  more 
seriously  to  rescue  scenario.  Resolving  this  issue  requires  not  only  improvement  of  each  robotic  and 
AI  agents,  but  also  how  such  systems  are  deployed  before  and  after  the  onset  of  the  disaster.  A 
systematic  approach  for  various  levels  of  information  fusion  is  necessary  for  robust  perception  of 
multi-agent  teams. 
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Chapter  2-  Biological  Robustness-  A  Basis  for 
Robust  Architecture 


The  architecture  described  in  chapter  1  is  a  result  of  investigation  on  biological 
robustness.  Basic  principles  identified  in  biological  system  have  been  reformulated  into 
engineering  system  framework. 

This  chapter  describes  biological  frameworks  that  supports  rational  behind  the 
proposed  architecture. 


2.1  Biological  Robustness 


Discovery  of  fundamental,  system-level  principles  behind  complex  biological  systems  is 
a  prime  scientific  goal  of  systems  biology  i  Robustness  is  a  property  of  the  system  to 
maintain  its  functions  despite  various  external  and  internal  perturbations.  It  is  one  of 
the  fundamental  and  ubiquitously  observed  phenomena  exhibited  at  the  system-level 
that  cannot  be  understood  by  looking  at  the  components  alone.  System  must  be  robust 
to  be  able  to  function  under  unpredictable  environment  using  sloppy  components. 
Understanding  the  origin  and  principles  of  robustness  in  biological  systems  will  help  us 
put  various  biological  phenomena  into  perspective;  it  will  also  catalyze  the  formation  of 
principles  at  the  system-level. 

In  this  article,  1  argue  that  robustness  is  a  fundamental  feature  of  evolvable  complex 
systems;  complex  biological  systems  shall  be  robust  against  environmental  and  genetic 
perturbations  to  be  evolvable,  and  evolution  often  selects  traits  that  may  enhance 
robustness  of  the  organism.  Thus,  robustness  is  ubiquitous  in  living  organisms  that 
have  evolved.  However,  systems  that  are  robust  face  fragility  and  performance  setback 
as  an  inherent  trade-off.  Identification  of  a  basic  architecture  for  a  robust  system  and 
trade-offs  embraced  is  essential  for  understanding  their  faults  and  countermeasures  — 
diseases  and  therapies. 


Robustness  as  an  organizational  principle 


Robustness  enables  the  system  to  maintain  its  functionalities  against  external  and 
internal  perturbations.  Such  a  property  has  been  widely  observed  across  species,  from 
gene  transcription  level  to  systemic  homeostasis  level.  For  example,  fate  decision  of 
lambda  phage,  which  either  lysis  or  lysogeny  pathways  has  to  be  activated,  was  once 
considered  a  result  of  fine  tuning  of  binding  affinity  of  promoters  and  corresponding 
regulatory  factors.  However,  it  was  shown  that  it  was  the  structure  of  the  network  that 
involves  both  positive  and  negative  feedback  that  is  responsible  for  making  sustainable 
commitment,  and  not  the  specific  binding  affinity,  therefore  the  fate  decision  behavior 
was  shown  to  be  robust  against  point  mutations  in  the  promoter  region  3.  In  addition, 
cooperative  binding  of  repressors  that  forms  implicit  local  positive  feedback  also 
contribute  to  stability  of  the  switch  Escherichia  coli  is  capable  of  maintaining 
chemotaxis  behavior  over  an  extremely  wide  range  of  chemo'attractant  concentration 
due  to  intracellular  integral  feedback  that  ensures  perfect  adaptation  that  is 
independent  of  ligand  concentration  ’^■®.  A  biochemical  network  involved  in  segmental 
polarity  in  Drosophila  has  been  shown  to  be  robust  against  changes  in  initial  values  and 
rate  constants  of  interactions  involved  that  enables  stable  pattern  formation  lo-n. 
Similar  observations  have  also  been  made  on  morphogen  patterns  formationsi°’i2,i3. 
Diseases  such  as  cancer  and  diabetes  are  manifestation  of  co'opted  robustness  in  which 
mechanisms  to  protect  our  body  are  effectively  taken-over  to  sustain  and  promote  the 
epidemic  states  i4-i6  There  are  many  more  examples  of  robust  properties  observed  in 
various  biological  systems.  As  studies  accumulate,  it  is  now  important  to  provide  an 
integrated  perspective  on  robustness  of  biological  systems. 

Robustness  of  a  system  may  manifest  itself  in  two  ways-  the  system  returns  to  current 
attractor  or  transit  to  a  new  attractor  that  ultimately  maintains  system’s  functions 
(Figure  l).  An  attractor  is  a  point  or  an  orbit  in  the  state  space  where  state  of  the  system 
asymptotically  converge.  A  return  to  the  current  attractor  is  often  called  “robust 
adaptation”.  The  attractor  can  be  either  static  (a  point  attractor;  a  fixed  point  in  the 
phase  space  in  which  trajectory  of  the  system  state  asymptotically  approaches)  or 
oscillatory  (a  periodic  attractor;  a  cyclic  orbit  in  the  phase  space  in  which  the  trajectory 
system  state  asymptotically  approaches).  A  transition  into  a  new  attractor  has  to  be 
made  robustly  in  response  to  stimuli,  so  that  the  system  behaves  consistently  against 
perturbations.  As  seen  in  lambda  phage  fate  decision,  stochastic  process  often 
influences  trajectory  of  transition  and  eventual  attractor  the  system  converge.  It  is 
often  misunderstood  that  robustness  is  to  stay  unchanged  regardless  of  stimuli  and 
mutations,  so  that  the  structure  and  components  of  the  system,  hence  state  of  operation. 


is  unaffected.  However,  robustness  actually  means  maintenance  of  specific 
functionalities  of  the  system  against  perturbations  and  it  often  requires  the  system  to 
change  its  state  of  operations  in  a  flexible  way.  In  the  other  word,  robustness  allows 
changes  in  the  structure  and  components  of  the  system  due  to  perturbations,  but 
specific  functions  are  maintained. 

In  the  following  sections,  I  will  outline  the  mechanisms  assuring  the  robust  operation  of 
a  system,  i.e.,  system  control,  alternative,  modularity  and  decoupling. 

System  control.  System  control  is  a  mechanism  that  consists  negative  and  positive 
feedback  to  attain  robust  dynamic  response  observed  in  a  broad  range  of  regulatory 
networks,  such  as  cell  cycle,  circadian  clock,  chemotaxis,  etc  Negative  feedback  is 

the  prime  mode  of  control  that  enables  robust  response  (or  robust  adaptation)  to  various 
perturbations.  Bacteria  chemotaxis  is  one  of  the  best  studied  example  of  robust 
adaptation  that  uses  negative  feedback,  an  integral  feedback  in  particular,  to  attain 
perfect  adaptation  that  enables  chemotaxis  against  broad  range  of  stimuli  '^'9.  It  was 
shown  that  particular  control  strategy,  an  integral  feedback,  is  essential  to  maintain 
robust  adaptation  in  both  E.  coli  and  Bacillus  subtilis  despite  the  fact  that  network 
topologies  are  not  identical  i®. 

Positive  feedback  contributes  to  robustness  by  amplifying  the  stimuli,  often  producing 
bi-stability,  so  that  the  activation  level  of  the  following  pathway  can  be  clearly 
distinguished  from  non-stimulated  states  and  such  state  can  be  maintained.  In 
Drosophila  segment  polarity  formation,  repetitive  stripes  of  differential  genes 
expressions  are  observed.  The  first  stripe  has  to  express  wingless  (wg),  the  second  stripe 
has  to  express  engrained  (en),  and  the  third  stripe  expresses  neither.  A  computational 
model  is  created  by  von  Dassow  and  colleagues  initially  without  positive  autoregulatory 
feedback  on  wg  and  en,  but  failed  to  reproduce  experimentally  observed  patterns. 
However,  with  two  positive  feedbacks  inserted  on  wg&Tid  en  activations,  robust  pattern 
formation  was  reproduced  i®.  Recently,  Ingolia  analyzed  this  model  and  showed  that  the 
bi-stability  caused  by  positive  feedback  loops  are  responsible  for  robust  pattern 
formation  n. 

Positive  feedback  is  also  used  in  signal  transduction  and  cell  cycle  to  form  switch-like 
behaviors  of  the  system  by  amplification  of  stimuli  as  well  as  for  fate  decision  as  seen  in 
the  lambda  phage,  so  that  it  initiatives  transition  and  a  new  state  of  the  system  is  made 
more  robust  against  noise  and  fluctuations  of  stimuh3>20-27  Many  biological  subsystems 
use  combination  of  such  system  control  to  attain  their  functions  and  its  robustness2'^-28_ 


Alternative,  or  failsafe  (Redundancy  and  Diversity).  Robustness  can  be  enhanced  if 
there  are  multiple  means  to  achieve  a  specific  function,  because  failure  of  one  of  the 
means  can  be  backed  up  by  other  means  available.  I  will  call  this  mechanism  as 
“alternative”,  or  “fail-safe”,  in  this  paper  (but  a  better  term  may  need  to  be  invented  to 
comfortably  represent  the  concept).  This  concept  includes  redundancy,  overlapping 
function,  and  diversity  with  the  difference  in  degree  of  similarity  of  available 
alternative  means.  Redundancy  generally  refers  to  situation  where  multiple  identical, 
or  similar,  components  (or  modules)  can  back-up  each  other  when  one  of  the  components 
failed.  Diversity,  or  heterogeneity,  represents  the  other  extreme  where  a  specific 
function  can  be  ultimately  attained  by  different  means  available  in  the  population  of 
heterogeneous  components.  Some  of  such  phenomena  have  been  well  recognized  as 
phenotypic  plasticity  29-3i_ 

In  some  tissues,  cells  may  be  surrounded  by  neighbors  that  are  similar  to  them,  so  that 
damaged  cells  are  quickly  replaced  by  other  cells.  But  having  multiple  identical 
components  as  alternatives  is  rare.  Alternative  is  mostly  attained  by  having  multiple 
heterogeneous  components  and  modules  with  overlapping  functions.  Recent  findings 
strongly  demonstrated  that  gene  duplication,  particularly  whole  genome  duplication 
followed  by  extensive  gene  loss  and  specialization,  is  one  of  critical  mechanisms  of 
evolutionary  innovation32,33^  providing  supports  for  long  standing  hypothesis  by 
Susumu  Ohno34.  While  duplicated  genes  are  diversified  by  mutation,  if  the  function  of 
duplicated  gene  pairs  partly  overlaps  it  act  as  an  evolutional  capacitor35,36^  and 
computational  study  indicates,  under  a  certain  condition,  such  pairs  could  be 
evolutionary  stables'?.  For  example,  budding  yeast  CLB5  and  CLB6  are  relatively 
homologous  genes  encoding  B-type  cyclins,  in  which  CLB6  shares  49.7%  identical 
residues  with  CLB5,  both  encoded  proteins  are  involved  in  S-phase  entry  of  cell  cyclers. 
Deletion  of  CLB6  has  no  or  little  effect  and  deletion  of  CLB5  has  prolonged  S-phase,  but 
deletion  of  both  genes  impedes  proper  timely  initiation  of  DNA  replication.  This  is  an 
example  of  genes  that  share  some  functions,  but  not  totally  identical. 

There  are  numerous  examples  of  alternatives  at  network-level,  too.  Text-book  examples 
include  glycolysis  and  oxidative  phosphorylation  39.  Although  both  processes  produce 
ATP,  the  oxidative  phosphorylation  requires  constant  supply  of  oxygen,  whereas 
glycolysis  can  be  either  aerobic  or  anaerobic,  although  the  latter  is  much  less  efficient. 
Diauxic  shift  in  yeast  causes  drastic  change  in  metabolic  pathways,  depending  on 
whether  glucose  or  ethanol  is  available  for  energy  metabolism  4o.  The  flux  balance  of 
metabolic  pathways  is  readjusted  depending  on  the  available  resources  in  the 


environment,  but  either  pathway  ultimately  serve  to  produce  essential  materials  for 
survival  and  growth  41,42  These  are  examples  of  phenotypic  plasticity  often  consider  to 
the  opposite  of  robustness.  However,  I  argue  it  is  more  consistent  to  view  phenotypic 
plasticity  as  a  part  of  robustness,  because  such  plasticity  enables  organisms  to  robustly 
adapt  to  varying  environment. 

It  is  interesting  to  note  here  that  while  redundancy  by  duplicated  genes  is  frequently 
observed,  where  is  no  reported  case  of  duplicated  circuits  despite  the  fact  that  there  are 
numbers  of  circuits  with  similar  topologies.  Investigation  of  networks  that  share  similar 
topology  as  far  as  the  degree  of  homologous  genes  involved  in  each  network  is  concerned 
revealed  that  while  genes  may  be  duplicated,  network  as  a  whole  is  not  duplicated  ^3. 
This  indicates  that  similar  network  topologies  in  different  places  within  and  between 
species  are  due  to  convergent  evolution,  rather  than  duplications  43.44  This  is  consistent 
with  the  fact  that  circuit  level  alternative  are  attained  by  different  implementations  of 
overlapping  functions. 

It  is  important  to  understand  that  alternative  is  coupled  with  system  control,  rather 
than  being  totally  independent.  First,  having  alternatives  at  components  level  enables 
regulatory  feedback  to  remain  intact  against  mutations.  Second,  switch  between 
alternative  means  has  to  be  orchestrated  by  specific  control,  so  that  system  behaviors 
are  properly  maintained. 


Modularity.  Modularity  is  an  effective  mechanism  to  locally  contain  perturbations  and 
damages,  so  that  the  effects  on  the  whole  system  can  be  minimized.  Module  is  widely 
observed  in  various  aspects  of  organisms  serving  as  a  possible  biological  design 
principles  30,45,46^  and  an  essential  element  in  engineering  and  industry  47.48  Despite 
intuitive  consensus,  the  concept  is  still  ambiguous  and  it  is  sometime  hard  to  detect46.  A 
cell  is  an  obvious  example  of  such  a  module  that  constitutes  multi-cellular  systems,  and 
it  interacts  with  environment  and  other  cells  mainly  through  receptors,  ion  channels, 
etc.  Modules  are  often  hierarchically  organized;  a  cell  itself  is  composed  of  organelles, 
such  as  mitochondria,  at  the  same  time,  it  is  also  a  part  of  larger  modules  such  as 
tissues  and  organs.  Aside  from  physical  modules  such  as  a  cell,  there  are  functional, 
spatial  and  temporal  modules  that  can  be  recognized  as  subsystems  of  metabolic 
network,  signal  transduction,  and  developmental  regulatory  networks.  Bacteria  flagella 
and  its  control  module  represents  both  physical  and  logical  module  that  is  robust  and 
versatile  40.  Although  such  modules  do  exist,  as  seen  in  segmental  polarity  networks  lon 
and  elsewhere  they  are  often  less  obvious  than  physically  partitioned  modules  and 


engineering  modules. 


Decoupling.  Decoupling  isolates  lowlevel  variation  from  high-level  functionalities.  For 
example,  Hsp90  not  only  fixing  proteins  that  are  mis-folded  due  to  environmental 
stresses,  hut  also  decouples  genetic  variations  from  phenotype  using  the  very  same 
mechanism,  thus  providing  the  genetic  buffer  against  mutations  Such  genetic 

buffers  decouple  the  genotype  from  the  phenotype  and  provide  robustness  to  cope  with 
mutation,  whilst  maintaining  a  degree  of  genetic  diversity.  These  buffers  have  been 
shown  to  underpin  the  robustness  of  developmental  processes  such  as  Waddington’s 
canalization54.  Importantly,  the  mutations  that  are  masked  by  genetic  buffering  are 
neutral  to  the  selection  (one  of  the  premise  of  Kimura’s  NEUTRAL  THEORY  OF 
EV0LUTI0N®5,56)^  thus  providing  a  source  of  material  for  the  evolution  of  the  system 
during  extreme  perturbations. 

Feedback  controls  sometimes  compensate  for  changes  in  kinetic  constants  and  initial 
values  of  networks,  as  seen  in  bacteria  chemotaxis'^  ®  and  Drosophila  segmentation!®,  or 
even  mitigate  the  impact  of  loss-of-function  mutant,  so  that  apparent  phenotype  may 
not  be  shown.  A  computational  study  on  cell  cycle,  originally  aimed  for  evaluating  model 
validity,  demonstrates  that  removing  some  genes  does  not  always  eliminate  cell  cycle, 
but  it  is  only  made  more  fragile  against  perturbations  Bistability  created  by  positivie 
feedback  sometimes  decouples  fluctuations  at  molecular  level,  such  as  numbers  of 
molecules  involved  in  reactions,  from  the  committed  state  of  the  system.  Thus, 
dynamics  of  networks  often  provides  decoupling  against  both  genetic  and 
environmental  perturbations®^.  There  are  even  a  hypothesis,  albeit  computational, 
claiming  such  buffering  is  an  intrinsic  nature  of  complex  networks  ss-eo 

Another  possible  decoupling  may  be  taking  place  between  information  encoding  and 
conversion  of  dosage  of  stimulus  into  pulse  of  protein  activations.  Upon  DNA  damage, 
p53-MDM2  feedback  loops  generate  oscillatory  behaviors,  and  such  a  behavior  was 
recently  found  to  be  a  potential  converter  of  graded  stimuli  (degree  of  DNA  damage)  to 
digital  pulse  (peak  of  p53  activation),  so  that  only  numbers  of  pulses  matters  after  the 
conversion 

The  mechanisms  behind  robustness  can  be  intuitively  understood  using  an  example  of  a 
sophisticated  engineering  object,  such  as  an  airplane  (Figure  2).  It  is  instructive  to 
know  that  very  similar  mechanisms  also  exists  in  a  variety  of  sophisticated  engineering 
systems  that  must  be  build  on  less  than  perfect  components  and  has  to  cope  with 


unpredictable  environment,  suggesting  universal  nature  of  robustness.  Interesting 
questions  here  are  whether  there  is  fundamental  system  architecture  for  successful 
robust  systems,  and  what  are  limitations  and  risks  of  associate  with  such  systems. 


The  Origin  of  Robustness 

It  is  now  increasingly  recognized  that  robustness  is  ubiquitous.  So  what  are  the 
principles  and  mechanisms  that  lead  to  the  emergence  of  robustness  in  biological 
systems?  My  conjecture  is  that  robustness  is  an  inherent  property  of  evolving  complex 
dynamical  systems,  meaning  various  mechanisms  incurring  robustness  of  organisms 
actually  facilitate  evolution,  and  evolution  selects  robust  traits.  Therefore, 
requirements  for  robustness  and  evolvability  are  almost  identical.  This  implies  that 
there  are  architectural  requirements  for  complex  systems  to  be  evolvable,  which 
essentially  requires  the  system  to  be  robust  against  environmental  and  genetic 
perturbations.  Robustness  is  ubiquitous  because  living  organisms  have  evolved. 

Evolvability  requires  flexibility  in  generating  diverse  phenotype  by  means  of  producing 
non-lethal  mutations  45,63,64  Kirschner  and  Gerhart  define  evolvability,  or  evolutionary 
adaptability,  as  a  capacity  to  generate  heritable  and  selectable  phenotypic  variations 
that  consists  of  features  that  “reduce  the  potential  lethality  of  mutations  and  the 
number  of  mutations  needed  to  produce  phenotypically  novel  traits”®^.  They  argue  that 
flexible  versatile  proteins,  weak  linkage,  exploratory  systems,  and 
compartmentalization  are  central  features  that  fosters  evolvability  63.  They  also  argue 
that  the  emerging  global  architecture  shall  be  composed  of  highly  conserved 
core-processes  that  are  co’selected  with  various  processes  some  of  which  brings 
phenotypically  novel  traits. 

Such  features  can  be  translated  into  architectural  requirements  of  the  system  that  is 
consistent  with  that  of  robust  systems.  First,  there  must  be  mechanisms  to  maintain 
components  and  interactions  intact  against  mutations,  yet  capable  of  generating  genetic 
variations.  Second,  there  must  be  modules  that  robustly  maintain  their  functions 
against  external  perturbations  and  mutations.  Third,  there  must  be  highly  conserved 
core-processes,  which  are  also  modular,  that  bear  fundamental  functions,  such  as 
metabolism,  cell  cycle,  and  transcriptional  machinery,  where  variety  of  modules  can  be 
interfaced  to  create  diverse  phenotypes.  The  overall  architecture  that  meets  such 


requirements  is  most  likely  to  be  a  modularized  nested  bowtie,  or  hour-glass,  structure 
where  various  inputs  and  outputs  modules  are  connected  to  conserved  core  (Figure  3),  a 
structure  also  identified  as  a  structure  of  the  Web^s.  Such  bowtie  structure  appears  is 
various  aspects  of  biological  systems  from  global  structure  to  specific  mechanisms  such 
as  transcription  and  translation  processes  The  architectural  features  entailed  in  this 
modularized  bowtie  structure  better  offers  robustness  to  the  various  aspects  of  system. 

Buffering 

The  first  step  in  robustness  is  to  maintain  components  and  interactions  intact  against 
perturbations.  As  already  discussed,  chaperons  including  hsp90  fix  proteins  misfolded 
due  to  external  stresses  such  as  heat  shock.  This  mechanism  also  applies  to  fixing  of 
proteins  that  are  genetically  varied,  but  expose  masked  genetic  variations  when  such 
mechanisms  are  impeded^s.  Networks  also  contribute  to  such  bufferings'^.  This  is 
particularly  the  case  when  it  is  robust  against  external  perturbations,  because 
regulatory  feedbacks  also  provides  robustness  to  perturbation  of  various  internal 
parameters,  as  already  discussed  in  chemotaxis'^  and  developmental  modulesio.  Genetic 
buffering,  also  called  evolutionary  capacitor,  either  attained  by  chaperons  or  network,  is 
one  of  the  fundamental  mechanisms  to  provide  both  robustness  and  evolvability.  There 
is  increasing  recognition  that  robustness  against  mutations  may  be  evolved  by 
congruence  that  is  side  effects  of  robustness  against  environmental  perturbations  and 
emergent  property  of  complex  networks®'^. 

Robust  Modules 

In  principle,  a  modular  system  enables  generation  of  diverse  phenotype  by 
rearrangements  of  its  inter-module  connections  and  relatively  independent  evolution  of 
each  module  through  mutations.  If  the  system  is  tightly  integrated  without  modular 
structure,  any  change  in  any  part  of  the  system  may  have  major  impact  on  other  parts 
of  the  system,  and  slight  changes  in  stimuli  or  noise  may  results  in  a  major  unexpected 
and  undesirable  outcome.  The  system  will  be  intractable,  and  it  is  very  difficult  for  such 
systems  to  generate  new  phenotypes  without  lethal  effects  —  the  system  is  too 
complicated  to  touch.  Modularization  significantly  mitigates  this  problem  by  enabling 
each  module  to  function  relatively  autonomously  from  others.  However  simply  having 
modules  is  not  enough  to  be  evolvable.  It  must  be  robust  against  various  perturbations 
such  as  changes  in  stimuli,  internal  kinetic  constants,  and  mutations.  This  feature  is 


essential  because  modules  need  to  be  able  to  cope  with  changes  in  stimuli  from  adjacent 
modules  that  might  evolve  somewhat  independently  or  function  in  the  different  context. 
Such  robustness  is  attained  by  system  control  and  alternatives  embraced  within  each 
module. 

The  contribution  of  system  control  to  the  module  robustness  is  particularly  important  in 
developmental  processes  where  novel  morphologies  can  be  explored  during  evolution. 
The  segment  polarity  network  in  Drosophila  was  shown  to  be  robust  against  variations 
in  kinetic  parameters  and  initial  concentrations  of  relevant  proteins,  as  long  as  central 
network  structure  is  maintainedio.  Similar  type  of  robustness  was  also  identified  in 
BMP  MORPHOGEN  gradient  formation's.  ORGANIZING  CENTRE  is  a  good  example  of 
robust  buffering  of  variation  in  developments^-™  A  portion  of  developmental  processes  is 
tolerant  against  variations  in  initial  values  and  robustly  forms  patterns  that  serve  as  a 
basis  for  further  elaborations  in  developmental  patterning,  amenable  to  reuse  under 
different  context '^1^2  fjox  genes  signifies  the  power  of  modularity  where  changes  in  Hox 
cluster  alone  affects  basic  body  plan  as  a  network  of  genes  downstream  of  the  gene 
develop  a  part  of  body  segment  relatively  autonomously  from  other  part  of  the  body'^3-75_ 
Mutations  in  Antenapedia  {antp)  and  eyeless  {ejd  exemplifies  master  control  genes  can 
ectopically  develop  appendages  and  being  nondethal  in  laboratory  environment^6-78^ 
although  viability  of  such  phenotype  in  natural  environment  is  questionable.  It  is 
important  to  recognize  that  nondethal  phenotype  does  not  means  such  phenotype  is 
favored  in  selection. 

Signal  transduction  pathways  play  important  role  in  phenotypical  diversity  during 
evolution.  Hedgehog  (Hh),  wingless  related  (Wnt),  TGF-beta,  receptor  tyrosine  kinase 
(RTK),  Notch,  JAK/STAT,  and  nuclear  hormone  pathways  are  signal  transduction 
pathways  widely  used  in  various  aspects  of  developmental  process.  Co-option  of  signal 
transduction  pathways  by  which  existing  pathways  are  used  for  different  processes  is 
considered  one  of  the  critical  features  in  evolution,  as  observed  in  Hh  signaling  pathway 
that  is  used  in  wing  pattern  formation  is  co-opted  in  butterfly  eyespots  pattern 
formation'll.  Numbers  of  signal  transduction  cascades  combined  both  negative  and 
positive  feedback  loops,  and  have  shown  to  be  robust  against  variations  of  stimuli  and 
kinetic  constants  so  that  it  can  maintain  normal  cellular  physiology  as  well  as 
developmental  processes  25,26,28,79,80_  This  intrinsic  robustness  of  the  pathway  enables 
co-option,  so  that  novel  morphologies  can  be  generated. 


The  origin  of  modularity  is  still  controversial.  It  is  certainly  an  evolved  property,  not 
necessarily  selectable  trait  by  itself,  and  it  enhances  flexible  generation  of  various 
phenotypes  in  development®!.  At  the  same  time,  modular  structures  and  modular 
regulatory  networks  within  a  single  cell  and  bacteria  that  demonstrate  enormous 
diversity  and  evolution  of  such  modules  indicate  flexibility  of  development  is  not  the 
only  reason  for  modularity.  Here,  I  would  discuss  two  possible  reasons  that  modularity 
initially  emerged  because  it  provides  robustness  against  environmental  perturbations. 
First,  emergence  of  modularity  of  gene  regulations  may  needed  handle  diverse  and 
complex  stimuli  and  responses^®.  It  is  essential  that  signaling  networks  and  reaction 
related  networks  are  modularized  to  some  extend  in  order  to  cope  with  various  external 
perturbations  without  mixing  up  stimuli  and  responses  relationship,  as  well  as  to 
prevent  effects  of  environmental  perturbations  to  spread  system-wide. 

Second,  if  modularized  phenotype,  in  both  morphology  and  regulatory  networks,  has 
selective  advantage  due  to  robustness  against  environmental  perturbations,  modular 
developmental  processes  may  be  co’selected  because  it  better  generates  modular 
phenotype.  This  hypothesis  assumes  that  modular  phenotype  has  selective  advantage 
and  modular  developmental  processes  better  generates  modular  phenotype;  neither  of 
them  yet  to  be  tested.  If  these  two  reasons  hold,  modularity  may  be  originated  to  bear 
robustness  against  environmental  perturbations,  but  congruent  with  flexibility  of 
development. 


The  Architectural  Framework 

There  are  two  architectural  features  that  facilitate  evolvability  and  robustness 
highly  conserved  core  processes  that  are  interfaced  with  diverse  inputs  (signaling  and 
nutrients)  and  outputs  (reactions  and  products)  and  versatile  mechanisms  that  bear 
essential  processes  of  the  system,  so  that  any  new  processes  that  properly  interface  with 
this  mechanism  can  utilize  such  common  mechanisms,  called  “weak  linkage”®®.  These 
features  represent  the  bowtie  architecture  in  different  scale;  a  global  topology  or 
specific  processes.  The  bowtie  structure  is  modular  and  nested,  so  that  it  appears  in 
various  aspects  of  biological  systems.  Here,  I  argue  that  bowtie  is  real,  robust,  and 
facilitate  evolvability. 


Bowtie  is  real 

Genome-wide  analysis  revealed  intriguing  characteristics  of  biological  networks  that 


support  bowtie  structure.  It  has  been  proposed  that  global  structure  of  metabolic 
networks  and  protein  interaction  networks  are  scale  free  network  created  by 
preferential  attachment82>83_  Scale  free  networks  are  tolerant  against  random  removal 
of  node,  but  fragile  against  systematic  removal  of  nodes  with  high  connectivity^^.  Ma 
and  Zeng  considered  directionality  of  reactions  in  the  analysis  of  metabolic  networks  of 
65  fully  sequenced  organisms,  and  found  that  the  overall  structure  of  the  network 
exhibits  a  ‘bowtie’  structure,  instead  of  scale  free  network,  in  which  a  big  highly 
connected  core  cluster  is  interfaced  with  less  connected  IN  and  OUT  clusters^s.  The  core 
of  the  network  is  a  giant  strong  component  (GSC)  sub-network  which  has  very  tight 
connections  among  its  components®^.  They  have  further  investigated  essential 
ingredients  of  GSC  across  species.  Comparison  of  the  metabolic  networks  between 
Streptococcus  pneumoniae  and  Pyrococcus  furiosus  demonstrates  conservation  of 
essential  metabolic  pathways,  such  as  the  TCA  cycle,  pentosephosphate  pathway,  and 
glycolysis  pathway,  within  GSC;  this  conserved  core  pathways  is  shown  to  be  robust 
against  perturbations  ®®. 

Another  indication  of  the  existence  of  such  conserved  processes  and  exploratory 
processes  comes  from  comparative  studies  using  functional  annotation  of  150  fully 
sequenced  genomes®®-®'^.  The  Gene  Ontology’s  biological  process  hierarchy®®  was  used  to 
annotate  functional  categories  to  each  gene,  and  proportions  of  numbers  of  genes  in 
each  functional  category  for  all  150  species  were  counted.  A  scaling  exponent  is  defined 
where  it  is  1.0  when  number  of  genes  in  a  category  doubled  when  total  number  of  genes 
doubled,  less  than  1.0  when  number  of  genes  in  a  category  is  less  than  rate  of  increase  of 
total  gene  number.  Cross  species  study  on  65  bacteria  genome  indicates  basic  biological 
processes  has  scaling  exponent  less  than  1.0  that  includes  cell  cycle  (0.47  +  0.08),  DNA 
repair  (  0.64  +  0.08  ),  DNA  replication  (  0.43  +  0.08  ),  and  protein  biosynthesis 
(0.13  +  0.02),  but  processes  that  may  generate  evolutionary  novelty  tend  to  have 
scaling  exponent  over  1.0  such  as  transcriptional  regulation  (1.87  +  0.13),  signal 
transduction  (1.72  +  0.18),  ion  transport  (  1.42  +  0.28  ),  two-component  systems 
(2.07  +  0.21),  and  cell  communication  (1.81  +  0.19)®®.  This  tendency  is  consistent  in 
extended  study  using  15  Archaea,  116  Bacteria,  and  10  Eukaryota  genomes®'^.  In 
addition,  scaling  exponent  for  genes  in  defense  response  category  in  Eukaryotes 
genomes  was  3.35  +  1.41,  highest  in  all  categories®®.  This  illustrates  that  there  are 
highly  conserved  processes  bearing  fundamental  biological  processes,  such  as 
biosynthesis  and  DNA  replications,  that  form  core  processes  and  weak  linkage,  as  well 
as  processes  added  with  increased  genome  size  possibly  bearing  generation  of  variety  of 


cell-types  and  morphological  features,  such  as  signal  transduction,  transcriptional 
regulation,  and  intercellular  communications.  This  implies  new  pathways  may  he 
constantly  added  to  wings  of  howtie  with  increase  of  genome  size. 

“Weak  linkage”  enables  addition  of  new  processes  to  the  existing  core  process  using 
common  mechanisms.  It  is  a  common  versatile  mechanism  that  operates  on  diverse 
inputs  and  outputs,  such  as  ion  channel,  G-protein,  and  transcription  machinery.  If 
transcription  machinery  is  different  for  each  gene,  addition  of  new  genes  and  new 
transcriptional  regulations  require  invention  of  customized  transcription  machinery 
that  makes  evolution  nearly  impossible.  GTP-binding  proteins  and  the  downstream 
cAMP  and  calcium  dynamics,  as  well  as  ion  channels  also  represent  systems  with 
common  underlying  mechanisms  that  allow  new  repertoire  to  be  added  64.89_  Various 
receptor  channels,  receptor  tyrosine  kinase  (RTK),  and  GPCR,  through  diverse 
mechanisms  and  adenylyl  cyclase  isoforms  converge  mainly  to  second  messengers,  such 
as  cAMP  and  calcium  ions,  that  mediates  variety  of  cellular  responses,  such  as  cell 
movement,  cell  growth,  metabolism,  etc  s®.  Cdc42,  a  Rho  family  of  GTPases,  is  another 
example  of  common  regulators  where  various  RTK  and  GPCR  pathways  converges,  and 
mediates  various  cellular  responses®^.  Most  signal  transduction  cascades  mainly 
converge  to  modulate  limited  numbers  of  second  messengers,  but  signaling  pathways 
are  often  diverse  and  cross-talked9i-®2  There  is  a  specific  pattern  of  interaction  that 
characterizes  logic  behind  it,  such  as  receptor  and  subunit  structures  of  G-protein, 
G-Protein  activation  by  ligand-bounded  receptors,  signaling  to  downstream,  etc.  This  is 
a  protocol®^,  so  that  diverse  signaling  pathway  can  be  created  by  numerous  isoforms  as 
far  as  this  protocol  is  kept,  and  contribute  to  diverse,  but  consistent,  signaling  leading  to 
weak  linkage  and  covered  core.  These  are  manifestation  of  bow-tie  structure  for  specific 
processes,  such  as  signal  transduction. 

Bow-Tie  is  robust 

Whether  bow-tie  structure  provides  robustness  against  external  perturbations  depends 
upon  robustness  of  the  conserved  core  and  global  regulations  imposed.  Ma  and  Zeng 
argues  that  GSC,  a  conserved  core  of  the  metabolic  network,  is  robust  against 
mutations  because  there  are  multiple  routes  between  any  pair  of  nodes  within  GSC®®. 
Variety  of  stimuli  activates  signal  transduction  pathways  converged  to  modulate  second 
messengers,  such  as  cAMP,  cGMP,  and  calcium  ions,  and  in  turn  activates  variety  of 
cellular  responses.  Here,  second  messengers  are  the  conserved  core  of  the  bow-tie 


architecture,  and  has  to  he  maintained  rohustly.  cAMP,  for  example,  is  produced  from 
adenylyl  cyclases  hy  ATP.  Interestingly,  there  is  a  variety  of  adenylyl  cyclase  isoforms 
involved  in  multiple  pathways  possibly  creating  alternative  pathways.  ATP  is  supplied 
from  a  robust  metabolic  core  mechanism  that  is  also  a  bowtie  structure.  Here,  the 
bow’tie  structure  at  metabolism  level  supports  the  bowtie  structure  at  signal 
transduction. 

In  addition  to  the  robustness  of  the  conserved  core,  the  bow  tie  architecture  may  provide 
advantage  in  generating  coordinated  response  to  variety  of  stimuli,  hence  improves 
robustness  against  external  perturbation,  by  having  variety  of  inputs  connecting  to  the 
robust  core  where  variety  of  reactions  are  mediated.  Direct  association  between  stimuli 
and  reactions,  without  using  the  robust  core,  requires  extensive  individual  controls  to 
attain  coordinated  response,  and  disruption  in  any  of  such  regulation  may  seriously 
undermine  system  behaviors.  Unless  each  stimuli  reaction  can  be  regarded  as 
independent,  so  that  coordination  is  not  required,  control  through  the  common  robust 
core  may  better  provides  robustness  of  the  system. 

One  might  ask  if  such  mechanisms  enable  to  accommodate  vast  diversity  of  possible 
phenotype.  In  fact,  versatile  mechanisms  are  essential  to  accommodate  variety  of 
possible  input  stimuli  and  reactions  in  a  consistent  manner,  because  addition  of  new 
signal  transductions,  for  example,  only  needs  to  be  interfaced  with  existing  machinery 
without  inventing  an  entire  cascade.  For  example,  recent  findings  of  differential  tissue 
and  celhtype  specific  expression  of  GPCRs  in  human  and  mouse®^  provide  explanations 
on  how  variety  of  cells  may  utilize  the  conserved  core  and  weak  linage  mechanisms. 
Expression  pattern  analyses  using  RT-PCT  on  100  randomly  selected  endoGPCR,  GPCR 
for  ligands  of  endogenous  origins,  from  392  mouse  endoGPCR  over  26  tissues,  17  from 
peripheral  tissues  and  9  from  different  brain  regions,  revealed  that  each  endoGPCR  is 
expressed  in  numbers  of  different  tissues,  but  forming  unique  combination  of 
endoGPCR  for  each  tissue^^.  This  suggest  a  possibility  a  relatively  small  set  of  second 
messengers  are  used  in  different  context  enabling  differentiation  into  variety  of  cell  tpes 
and  a  range  of  cellular  responses.  Same  argument  also  applies  to  toolkit'^2^  that  is 
limited  set  of  versatile  genetic  networks,  because  they  can  be  used  in  different  context. 

Conservation  of  Robustness 


Since  robustness  is  ubiquitous  and  inherent  feature  of  biological  systems,  it  is 


important  to  understand  intrinsic  dynamics  of  such  systems.  It  is  interesting  to  ask 
whether  there  is  principle  of  conservation  and  symmetry  as  seen  in  physics.  I  suggest 
that  there  may  he  similar  principles  for  robustness  of  the  biological  systems,  although 
they  could  require  representation  that  is  not  as  simple  as  their  counterparts  in  physics. 
Here,  I  will  discuss  theoretical  properties  of  such  systems. 

I  have  argued  that  robustness  is  a  fundamental  feature  that  enables  complex  systems  to 
evolve,  and  evolution  enhances  robustness  of  organisms.  One  possible  path  following 
this  scenario  is  increasing  complexity  of  organisms  by  successive  addition  of  regulatory 
systems,  such  as  diverse  regulations,  signal  transduction  pathways,  RNA 
regulationsS®'^'^,  and  histone  modifications®®,  to  enhance  robustness  against  specific 
environmental  perturbations  and  exploration  into  un-occupied  niche®®.  However, 
introduction  of  various  control  loops  generates  tradeoff  by  causing  instability  when 
unexpected  perturbations  are  encountered,  leading  to  catastrophic  failure.  It  is  widely 
accepted  in  the  control  theory,  that  increased  robustness  against  non-linearity  and  noise 
using  negative  feedback  control  is  associated  with  instability  elsewhere.  Carlson  and 
Doyle  tried  to  generalize  such  issues  in  their  highly  optimized  tolerance  (HOT)  theory 
by  arguing  that  systems  that  are  evolved  to  be  robust  against  general  perturbations  are 
extremely  fragile  against  certain  types  of  rare  perturbations^®®’!®!.  Using  simplified 
models  in  physics,  forest  fire,  etc,  they  argued  that  systems  that  are  successively 
optimized  to  certain  type  of  perturbations  develop  a  characteristic  structure  which  is 
efficient  and  robust  to  cope  with  such  perturbations,  but  could  be  extremely  fragile 
against  unexpected  perturbations.  In  dynamical  systems,  such  evolutionary 
optimizations  are  done  by  successively  adding  feedback  controls  to  the  system.  It  is 
important  to  recognize  that  while  HOT  describes  behaviors  of  complex  systems  designed 
or  evolved  toward  optimality  (while  it  may  be  suboptimal)  against  perturbations  (like 
modern  airplanes  and  biological  systems),  other  self-organization  models  such  as  scale 
free  network®®’!®^  by  preferential  attachment  and  self  organized  criticality  (SOC)!®® 
models  systems  with  stochastic  additions  of  complexity  without  design  or  evolution 
involved  (like  sand  piles).  Nature  of  robustness  and  fragility  predicted  by  HOT  model  is 
generally  consistent  with  observed  properties  of  designed  complex  systems  and  sharply 
different  from  other  model!®®!®!.  Csete  and  Doyle  further  argued  that 
robustness/fragility  trade-offs  indicate  that  robustness  is  a  conserved  quantity,  a 
concept  that  also  applies  to  biological  systems®®. 

Take  an  airplane  example  again.  The  Wright  brothers’  airplane  is  not  robust  against 
atmospheric  perturbations,  whereas  modern  commercial  airplanes  are  extremely  robust 


against  such  perturbations.  However,  modern  airplanes  are  extremely  fragile  against 
very  unusual  perturbation  such  as  total  power  failure  because  their  flight  control 
system  is  totally  dependent  on  electricity  (obviously,  all  possible  counter  measures  are 
taken  to  prevent  such  failures).  Wright  flyer,  on  the  other  hand,  has  no  impact  on  such 
failure  as  it  does  not  use  electric  control  in  the  first  place.  It  is  important  to  understand 
such  trade-offs  because  diseases  are  often  manifestation  of  such  fragility,  as  I  will 
discuss  later. 

Fragility  is  not  the  only  cost  of  the  improved  robustness.  In  electronic  circuit  design,  the 
use  of  negative  feedback  control  attains  improved  response  against  a  certain  range  of 
inputs  by  sacrificing  an  overall  gain  of  the  amplifier.  Thus,  the  use  of  negative  feedback 
control  gains  robustness  within  a  certain  range  of  inputs  at  the  cost  of  an  aspect  of 
performance  and  fragility  elsewhere.  Such  trade-off  between  robustness,  fragility,  and 
performance  can  be  widely  observed  in  biological  systems  at  different  levels. 

Having  alternatives  and  modularity  also  enhances  robustness,  but  at  the  cost  of 
increased  resource  demands.  For  example,  a  probability  of  a  function  to  fail  with  a 
single  component  to  attain  this  function  is  P,  where  P  is  a  probability  of  failure  of  the 
component.  A  probability  of  the  function  to  fail  with  two  components  to  back  up  each 

other  will  be  reduced  10(1  —  /*)^.  However,  resource  requirement  is  doubled.  Such  a 

trade-off  is  effectively  mitigated  in  biological  systems  because  components  involved  tend 
to  have  overlapping  functions,  instead  of  being  identical  copies.  Having  identical  copies 
as  alternatives  is  only  efficient  when  failure  rate,  or  turn  over  rate,  is  expected  to  be 
very  high.  Modularity  also  trade-off  between  robustness,  flexibility,  and  resource 
demands.  Merging  modules  to  share  common  circuits  and  components  reduces  resource 
needs,  but  robustness  in  preventing  spread  of  perturbation  and  flexibility  of 
rearrangement  are  seriously  compromised. 

Although  several  trade-off  exists  in  robust  complex  systems,  trade  off  involving  system 
control  is  the  most  significant  one.  It  defines  dynamical  properties  of  the  biological 
systems  which  illustrate  how  complex  biological  systems  behave  when  perturbed,  as 
well  as  orchestrates  how  alternative  components  and  modules  are  re-routed  to  ensure 
robustness  of  the  whole  system.  Due  to  intrinsic  trade-offs  discussed  above,  it  is  not 
possible  to  simply  increase  general  robustness  of  the  system  without  sacrifice  of 
performance  and  increased  resource  demands. 


Co-opted  robustness:  system-level  view  of  disease  and  therapy 


Theoretically  underlined  properties  of  robust  evolvable  systems  have  direct 
consequences  in  our  understanding  of  diseases  and  therapy  design.  First,  robust 
systems,  whether  biologic  or  engineering,  are  most  vulnerable  when  its  fragility  is 
exposed.  Diabetes  mellitus,  for  example,  can  be  considered  as  an  exposed  fragility  of  the 
system  that  has  acquired  robustness  against  near-starvation,  high  energy  utilization 
life  style  and  high  risk  of  infection,  but  unusually  perturbed  by  over-nutrition  and  low 
energy  utilization  lifestyle  i®.  Second,  the  system  is  relatively  tolerant  with  simple 
“faikoff’  of  components,  such  as  removal  of  some  components  or  cells,  due  to 
alternatives  available  and  as  intrinsic  nature  of  the  bow-tie  architecture.  However,  the 
system  is  vulnerable  is  when  components  “faikon”  where  components  are  not  being 
removed,  but  behave  inappropriately.  In  engineering  field,  such  failure  is  known  as  the 
Byzantine  Generals  Problems,  named  after  the  problem  of  the  Byzantine  army  with 
traitors  among  own  generals  who  send  confusing  messages,  that  has  not  yet  fully 
solvedi®4.  Third,  epidemic  state  may  exhibit  robustness  against  natural  and  therapeutic 
countermeasures  if  intrinsic  mechanisms  for  robustness  of  our  body  are  co-opted.  The 
worst  scenario  is  that  fragility  is  exposed  that  triggers  outbreak  of  disease  which  causes 
faikon  failure,  and  such  faikon  failure  persists  by  taking  over  intrinsic  robustness  of  our 
body,  so  that  an  epidemic  state  is  locked-in. 

This  view,  which  is  theoretically  motivated,  is  particularly  important  because  intrinsic 
nature  of  diseases  and  appropriate  countermeasures  are  different  depends  on  which 
scenario  the  system  failure  follows.  For  example,  cholera  toxin  interacts  with  Gs-alpha 
submit  to  trigger  the  symptoms^®®’!®®.  However,  it  can  be  simply  removed  by  antibiotics, 
because  robustness  has  not  been  hijacked.  Thus,  typical  counter  measures  involve 
preventing  a  cascade  of  failures  by  supplying  waters  to  prevent  excessive  loss  of  liquid 
and  using  antibiotics  (tetracycline)  to  remove  pathogens  by  inhibiting  protein 
production. 

However,  cancer  and  HIV  represents  worst  scenario  that  epidemic  state  are  maintained 
and  even  promoted  because  of  intrinsic  robustness  mechanisms  of  host  system.  Counter 
measures  for  such  diseases  may  beJ  (l)  to  actively  perturb  specific  interactions  or 
components  to  maintain  or  reduce  robustness,  (2)  to  find  a  point  of  fragility  “Achilleus’s 
heel”  that  is  inherent  in  robust  systems,  and  (3)  to  retake  control  of  epidemic  state  by 
introducing  counter-acting  decoy  “Trojan  horse”  or  a  new  regulatory  feedback. 


Cancer  represents  nightmare  scenario  where  faikon  components,  tumour  cells,  acquired 


robustness  against  natural  defense  and  various  therapeutic  interventions.  Cancer  is  a 
highly  robust  disease  in  which  tumour  proliferates  and  metastasizes,  in  some  cases 
despite  much  therapeutic  efforts.  Although  anti-cancer  drugs  may  temporarily  reduce 
tumour  mass,  it  relapses  in  many  cases  and  cure  is  still  rare.  The  difficulty  of  treating 
tumours  is  due  to  acquired  robustness,  partly  as  a  result  of  co-opting  intrinsic 
mechanisms  for  robustness  High  level  of  genetic  heterogeneity  in  the  tumour  that 
forms  heterogeneous  redundancy  and  multiple  feedback  loops  in  both  cellular  and 
tumour-host  environment  account  for  robustness  of  cancer.  Genetic  heterogeneity 
within  a  tumour  cluster  due  to  chromosome  instability  generates  high  level  of 
heterogeneous  redundancy  in  a  function  to  survive  and  proliferate.  This  heterogeneous 
redundancy  allows  some  malignant  cells  to  tolerate  a  therapy  and  reform  a  tumour. 
Mechanisms  that  maintain  normal  functions  of  the  body  also  function  to  enhance 
tumour’s  robustness  against  therapy.  For  example,  drug  resistance  is  caused  by 
up-regulation  of  MDRl  and  other  genes  that  pump  out  toxic  chemicals  from  the  cell;  so 
a  function  that  protects  us  in  normal  condition,  is  exploited  in  tumour  to  protect 
malignant  cells.  A  low  oxygen  supply  (hypoxia)  caused  during  the  tumour  cluster 
development  is  countered  by  metabolic  shift  from  oxygen-dependent  TCA  cycle  to 
glycolysis,  as  well  as  activating  feedback  loop  by  up-regulating  HIFl,  which 
up-regulates  VEGF  to  promote  angiogenesis,  and  MMP,  uPAR,  and  CRCX4  that 
promote  tumour  cell  metastasis  1°'^,  a  series  of  responses  that  protects  us  in  general  low 
oxygen  conditions. 

From  the  robustness  perspective,  possible  clinical  strategies  against  cancer  are;  to 
control  robustness  of  tumour,  and  to  finding  out  the  point  of  fragility  that  is  inherent  in 
the  robust  system.  To  control  robustness,  therapy  should  be  directed  to  induce  tumour 
dormancy  by  selectively  inducing  cell  cycle  arrest,  rather  than  aiming  at  tumour 
eradication,  because  one  source  of  robustness  is  genetic  heterogeneity  created  through 
somatic  recombination.  Apart  of  specific  cases  where  tumour  cells  are  relatively 
homogeneous  so  that  drugs  that  targets  specific  molecule  can  have  dramatic  effects, 
tumour  mass  reduction  may  not  be  an  appropriate  therapeutic  goal,  because  of  a  high 
risk  of  relapse  if  the  reduced  tumour  gained  greater  level  of  heterogeneity  that  includes 
mutant  cells  highly  resistant  against  therapeutic  efforts.  Controlling  tumour  robustness 
would  be  a  genuine  measure  of  therapeutic  efficacy,  so  that  the  risk  of  relapse  can  be 
well  controlled.  The  other  approach  may  be  to  find  fragility  of  tumour.  Since  tumours 
undergo  clonal  evolution  and  acquire  robustness  against  a  range  of  therapies,  the 
principle  of  conservation  of  robustness  indicates  there  must  be  the  point  of  extreme 
fragility.  Therefore,  efforts  to  find  fundamental  fragility  of  tumour  for  therapeutic 


targets  should  be  made  in  future. 

The  question  is  how  to  find  fragility  that  is  therapeutically  effective  and  practical.  It  is 
important  here  to  investigate  what  system  has  been  optimized  for  and  identify  sources 
of  robustness  epidemic  acquired.  Since  fragility  is  a  by-product  of  robustness  elsewhere, 
fragile  point  of  the  system  must  be  associated  with  mechanism  that  gave  rise  to 
enhanced  robustness.  For  example,  robustness  of  tumour  is  sustained  by  chromosome 
instability,  intracellular  feedback  loops,  and  host-tumour  interactions. 
Countermeasures  to  control  cell  cycle  by  multiple  molecular  targets  possibly  using 
RNAi,  selectively  de-stabilize  instable  chromosome,  infiltrate  genetically  engineered 
tumour  associated  macrophage^o®  to  retake  control  of  host-tumour  interactions,  and 
introduce  artificial  genetic  circuits^o®  to  conditionally  express  tumour  suppressor  genes 
can  be  potential  candidates,  but  has  to  be  designed  carefully  to  specifically  explore 
frgility  or  to  control  robustness. 

HIV  infection  is  the  other  worst  case  scenario.  HIV  predominantly  infects  CD4-I-  T  cell 
and  replicate  when  CD4-I-  T  cell  activate  its  anti-virus  responses  no-us.  Infected  cells  are 
not  removed  because  infection  is  hardly  detectable.  This  is  typical  case  of  faihon 
problem.  In  addition,  HIV  creates  highly  diverse  genetic  alternation.  This  HIV’s 
strategy  is  a  hijack  of  robust  immune  response  mechanism,  which  makes  AIDS  difficult 
to  cure.  Robust  persistence  of  epidemic  state  through  generation  of  genetic  diversity  and 
various  feedback  loops  are  similar  to  the  strategy  found  in  cancer.  Possible  therapeutic 
approaches  discussed  already  may  apply  for  effective  treatment.  Interestingly  enough, 
one  strategies  proposed  to  combat  HIV  involves  putting  HIV  into  latent  state,  instead  of 
trying  to  remove  it,  by  introducing  the  decoy  that  is  conditionally  replicating  HIV-1 
vector  (crHIV-l)  114-116.  crHIV-1  contains  only  cis,  but  not  trans  elements  that  are 
necessary  for  virus  packaging,  and  carries  antiviral  genes  that  inhibits  wild  type  HIV-1 
functions!!®.  If  this  is  successful,  it  may  retake  control  of  the  system,  and  put  HIV-1 
virus  in  prolonged  latency. 


Toward  the  theory  of  biological  robustness 

Given  the  importance  of  robustness  for  understanding  of  principle  of  life  and  medical 
implications,  it  is  an  intriguing  challenge  to  formulate  mathematically  solid,  and 
possibly  unified  theory  of  biological  robustness  that  may  serve  as  a  basic  organizational 
principle  of  biological  systems;  some  early  attempts  date  back  to  the  middle  of  the  last 


century  ii7.ii8_  Such  a  unified  theory  could  he  a  bridge  between  fundamental  principles 
of  life,  medical  practice,  engineering,  physics  and  chemistry.  This  is  a  major  challenge  in 
which  numbers  of  issues  that  has  to  be  solved  particularly  to  establish  mathematically 
well'founded  theories,  but  the  impact  would  be  enormous. 

First,  solid  quantitative  index  of  the  robustness  has  yet  to  be  established.  Such  index 
has  to  be  able  to  equate  with  experimentally  measurable  quantities,  so  that  hypotheses 
on  the  degree  of  robustness  can  be  tested  experimentally.  There  are  several  concepts  on 
STABILITY  OF  THE  SYSTEM  in  CONTROL  THEORYHs,  and  some  attempt  has  been  made  to 
apply  such  ideas  to  biological  robustnessi^o.  In  non-linear  dynamics  theory,  there  is  a 
concept  of  lyapunov  stability  that  represents  system’s  tendency  to  asymptotically  return 
to  the  attractor.  However,  practical  application  of  these  concepts  to  biological  systems 
has  been  limited,  mainly  due  to  an  enormous  degree  of  dimensionality  and  non-linearity 
that  are  intrinsic  to  biological  systems.  Measuring  perturbations  for  all  such  dimensions 
is  impractical.  Practical,  yet  theoretically  solid,  index  needs  to  be  developed  to  provide 
guidelines  for  selecting  smaller  numbers  of  parameters  to  be  perturbed  so  that  the 
system’s  response  can  be  measured.  Haken  pointed  out  that  only  a  few  parameters  are 
important  to  describe  system  behavior  near  bifurcation  point  121,  but  this  approach  is  too 
limited  to  analyze  various  behaviors  of  biological  systems.  Efforts  are  undergoing  to 
develop  a  mathematical  method  to  derive  a  set  of  inequities  of  parameters  that  ensure 
stability  of  the  systemi22,i23_  While  it  is  in  infancy  and  has  only  limited  applicability  now, 
further  progress  in  this  area  would  certainly  benefits  mathematical  analysis  of 
robustness  of  biological  systems.  In  experimental  front,  comprehensive  mutant  creation 
and  dosage  controlled  perturbations  are  already  practical  for  some  organisms.  Since, 
there  are  often  practical  aims  to  investigate  robustness  of  the  system  such  as  to  induce 
cell  cycle  arrest,  experiments  can  be  carefully  designed  to  perturb  relevant  genes  and 
parameters,  and  published  data  can  be  systematically  collected,  to  obtain  practically 
sufficient  index  of  robustness  for  specific  aspects  of  the  organismi24. 

Second,  the  theory  that  embraces  various  aspects  of  robustness  has  not  yet  been 
formulated.  Control  theory  is  often  used  to  explain  robustness  that  involves  feedback 
regulation,  but  it  only  covers  one  aspect  of  robustness  that  involves  negative  feedback 
control.  Furthermore,  the  control  theory  assumes  that  there  is  a  certain  set  point  that 
the  system’s  state  shall  approach,  even  under  perturbations,  and  that  is  assumed  to  be 
determined  by  the  designer.  Of  course,  there  is  no  such  designer  in  biological  systems, 
and  set  point  is  implicit  in  the  equilibrium  state  of  the  system,  and  such  equilibrium 


often  changes  dynamically.  Theory  need  to  he  formulated  to  reflect  such  features  of 
biological  systems. 

Recent  efforts  to  integrate  control  theory  and  Shannon’s  channel  coding  theorem  may 
provide  interesting  framework  for  feedhacksi^s.ise  This  theory  tries  to  formalize  cases 
where  there  is  limitation  of  information  capacity  in  feedback  loop,  so  that  feedback 
signal  is  potentially  impeded  on  fidelity,  noise,  and  delay.  This  better  reflects  reality  of 
biological  systems  in  which  signals  are  transmitted  with  noise,  delay,  and  compromised 
fidelity. 


Third,  challenges  to  relate  dynamics  of  life  and  thermodynamics  have  yielded  only  a 
limited  success.  Behaviors  of  the  physical  and  chemical  systems  near  equilibrium  have 
been  well  investigated  in  the  past.  Le  Chatelier-Braun’S  PRINCIPLE  provides  a  basis 
for  response  against  perturbations  for  systems  in  equilibrium,  as  a  special  form  of  more 
general  principle  known  as  THEOREM  OF  MODERATION127.  The  principle  indicates 
emergence  of  compensatory  feedback  to  cope  with  perturbations.  Efforts  have  been 
made  to  extend  thermodynamics  for  far  from  equilibrium  to  explain  biological 
phenomenai28-i30^  but  applied  only  for  relatively  simple  chemical  reactions  under 
isotropic  medium  such  as  Belousov-Zhabotinski  (BZ)  REACTIONS.  Signal  transduction, 
for  example,  undergoes  dramatic  changes  in  the  state  of  the  cell  depending  upon  the 
intensity  and  types  of  stimuli,  resembling  transition  from  near  equilibrium  state  to  far 
from  equilibrium  state.  But,  this  is  not  due  to  simple  chemical  reactions  under 
unstructured  free  medium  as  seen  in  BZ  reactions.  Cells  are  highly  structured  and  have 
explicit  interactions  and  physical  structures  controlled  by  gene  expressions, 
cytoskeleton  and  other  regulatory  systems  that  are  optimized  to  be  robust  and  evolvable. 
This  contrasts  with  much  favored  toy  examples  of  simple  chemical  reactions  or  sand 
piles  that  are  not  structured.  The  major  challenge  is  to  formulate  theories  that  accounts 
for  thermodynamics  in  nonusotropic,  heterogeneous,  and  structured  system,  because 
such  well  established  mathematical  frameworks  for  describing  such  systems  are  not 
available  yet.  Some  efforts  are  being  made  to  find  such  a  theoretical  framework  by 
assuming  network  structure  as  a  basis  of  mathematical  descriptioni^i.  However,  it  is 
still  to  abstract  to  be  practically  applied  for  biological  systems.  Progress  in  this  arena 
will  help  closely  connect  biology,  chemistry,  physics,  and  mathematics  in  a  coherent 
manner. 


Conclusion 


Robustness  is  a  fundamental  property  of  biological  systems.  It  facilitates  evolvability, 
and  evolution  selects  robust  traits.  1  have  argued  that  there  are  specific  architectural 
features  for  organisms  to  be  robust  that  may  be  universal  to  any  robust  and  evolvable 
complex  systems.  Systems  control,  modularity,  alternative,  and  decoupling  serve  as 
basic  mechanisms  to  provide  robustness  to  the  system,  but  need  to  be  organized  into 
coherent  architecture  to  he  effective  at  organism-level.  Enhancement  of  robustness 
against  perturbations  can  be  made  with  combination  of  such  mechanisms,  but  systems 
control  is  the  prime  mechanism  for  coping  with  environmental  perturbations  that 
requires  proper  dynamics.  Thus,  evolution  of  organisms  can  be  viewed,  at  least  in  one 
aspect,  as  evolution  of  control  systems.  Modularity,  alternative,  and  decoupling,  in  part, 
support  robust  maintenance  of  such  control  loops,  but  are  also  controlled  by  such  control 
loops  either  explicitly  or  implicitly.  There  is  an  intriguing  possibility  that  genetic 
buffering  and  modularity  originated  from  robustness  against  environmental 
perturbations,  and  evolved  to  have  wider  applicability.  It  is  important  to  realize  that 
systems  that  are  evolved  to  be  robust  against  certain  perturbations  are  extremely 
fragile  to  unexpected  perturbations.  This  robust  yet  fragile  trade-off  is  fundamental  to 
complex  dynamical  systems.  Importance  of  understanding  architectural  features  of 
robust  and  evolvable  system,  and  intrinsic  nature  of  robustness  and  fragility  embraced 
is  that  it  dictates  mode  of  system  failures  and  effective  counter  measures  -"  diseases 
and  therapies.  Failures  and  viable  counter-measures  of  such  systems  are  often 
counter-intuitive,  implies  theoretically  motivated  robustness  perspective  may  provide 
novel  therapeutic  approaches.  The  emerging  field  of  systems  biology  has  been  trying  to 
identify  system-level  properties,  but  simple  use  of  massive  data  and  computation  would 
not  effectively  reveal  insights  for  biological  system  and  applications  for  medicine.  The 
perspective  on  biological  robustness  would  provide  effective  guiding  principles  for 
understanding  various  biological  phenomena  and  design  of  therapies. 
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FIGURE  i:  Robust  reactions  of  the  system:  to  stay  or  to  transit 

State  of  systems  can  be  shown  as  a  point  in  the  state  space.  For  the  sake  of  explanation, 
this  figure  simplifies  the  state  space  into  only  two  dimensions.  Perturbations  forcefully 
move  such  a  point  of  system’s  state.  State  of  the  system  may  return  to  its  original 
attractor  by  adapting  to  perturbations,  often  using  negative  feedback  loop.  Bacterial 
chemotaxis  is  a  noteworthy  example.  There  are  basins  of  attractions  in  the  state  space 
where  state  of  the  system  moves  back  to  the  attractor.  If  perturbation  exceeds  boundary, 
the  system  may  move  into  an  unstable  region  or  transit  to  other  attractors.  Positive 
feedback  have  dual  role  in  moving  systems  state  away  from  the  current  attractor,  or  to 
push  system  into  the  extreme  when  there  is  an  attractor  for  such  a  state.  Cell  cycle  is  a 
combination  of  positive  and  negative  feedbacks  that  facilitate  transition  between  two 
attractors  (Gl  and  S/G2/M)  forming  a  bi-stability.  Often  stochastic  processes  affect  a 
transition  between  attractors,  as  seen  in  lambda  phage  fate  decision,  but  maintenance 
of  new  state  has  to  be  robust  against  minor  perturbations. 

FIGURE  2:  Explaining  robustness  by  airplane 

The  concept  of  robustness  is  best  described  using  the  example  of  robustness  of  modern 
airplanes.  Many  commercial  passenger  airplanes  have  an  automatic  flight  control 
system  (AFCS)  that  maintains  a  flight  path  (direction,  altitude,  and  velocity  of  flight) 
against  perturbations  in  atmospheric  conditions.  This  can  be  accomplished  by  feedback 
control  in  which  deviations  from  the  defined  flight  path  are  automatically  corrected. 
AFCS  is  the  critical  component  that  enables  robust  maintenance  of  flight  path  by 
controlling  airplane’s  flight  control  surfaces  (rudder,  elevator,  flaps,  aileron,  etc)  and  a 
propulsion  system  (engines).  AFCS  is  generally  composed  of  three  modules  with  same 
functions  thereby  maintaining  redundancy,  but  each  designed  differently 
(heterogeneity)  to  avoid  a  common  mode  failure.  Three  computers  are  made  modular,  so 
that  failure  in  one  module  does  not  affect  functions  of  other  part  of  the  system.  Such 
system  is  implemented  using  digital  technologies  that  decouple  lowlevel  voltages  from 
digital  signal  (ON/OFF  of  pulses),  thereby  rejecting  noise  to  influence  its  functions. 
Although  this  is  a  very  simplified  explanation  of  the  actual  system,  the  basic  concept 
applies  to  details  of  the  system  as  well  as  to  more  complex  systems.  Although 
differences  between  man-made  systems  and  biological  systems  exist,  similarities  are 
overwhelming.  Fundamentally,  robustness  is  the  basic  organizational  principle  of 


evolving  dynamic  systems,  be  it  through  evolution  or  competition  and  niche  in  the 
market  and  society. 

FIGURE  3:  The  architectural  framework  of  robust  evolvable  systems. 

The  bow-tie  (or  hour-glass)  structure  has  diverse  input  and  output  that  are  connected 
through  a  conserved  core  and  versatile  weak  linkage  with  extensive  system  control 
governing  the  dynamics.  Core  processes  and  versatile  interfaces  overlap  or  merge  in 
some  cases.  This  bow-tie  structure  appears  at  various  level  of  the  system,  such  as 
metabolism,  signal  transduction,  transcription,  and  translation®®.  In  signal 
transduction,  diverse  stimuli  are  initially  received  by  receptors,  signaling  pathways, 
including  diverse  isoform  of  G-proteins,  are  activated,  but  converge  mainly  to  second 
messengers  that  have  limited  variety  and  serve  as  weak  linkage.  Then,  modulations  in 
second  messengers  influence  core  processes  to  trigger  differential  gene  expressions  and 
diverse  reactions.  However,  this  process  is  not  a  simple  flow  as  extensive  local  and 
global  feedback  regulations  are  imposed  in  every  step.  Metabolism  takes  diverse 
nutrients,  preprocess  them  into  precursors  in  which  core  metabolic  pathways  covert 
them  into  basic  currencies  such  as  ATP  and  NADH,  and  well  as  activating  biosynthesis 
pathways  to  produce  amino  acid,  nucleotide,  sugar,  etc.  Transcription  and  translation 
also  represent  such  structures  where  common  machineries  are  used  to  decode  a  wide 
range  of  genetic  information  and  produces  diverse  proteins,  but  versatile  mechanisms 
themselves  are  conserved  core.  Various  processes  are  interfaced  with  core  processes 
through  versatile  interfaces,  so  that  novel  processes  can  be  added  and  removed  easily 
without  seriously  affecting  other  parts  of  the  system. 
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Chapter  1 

Overview  of  the  Modular 
Sensor  System 


A  sensory  system  consists  of  many  sensors  and  often  some  actuators  because 
the  more  sensor  there  is  means  more  information  to  be  acquired.  However,  the 
handling  of  the  information  among  different  dimension  makes  it  difficult  to  be 
an  open  system  because  information  stream  from  the  sensor  is  so  different  that 
makes  it  difficult  to  provide  the  unitary  interface  between  them. 

The  proposed  system  resolved  this  problem  by  constructing  the  system  dis¬ 
tributed  intelligent  telemetry  and  centralized  the  decision  making.  The  system 
is  designed  to  be  modular  in  both  macro  and  micro  structure.  Since  the  system 
is  modular  in  both  hardware  and  software,  it  is  expandable  and  configurable. 
For  this  development,  we  applied  this  system  on  finding  and  tracking  the  sus¬ 
picious  object  in  closed  area. 

The  macro  structure  of  the  system  is  based  on  IP  network.  The  sensor  device 
and  system  controller  is  connected  with  100Base-TX  Ethernet.  The  information 
through  the  network  is  as  follows. 

•  Control  signal  (from  controller) 

•  POI  (Point  of  Interest)  information  (from  sensors) 

•  Telemetric  data  (from  the  sensor,  on  demand) 

,  Control  signals  and  POI  data  were  transmitted  over  UDP/IP  to  minimize 
the  overhead  of  transmission,  but  TCP/IP  can  also  be  used  in  situation  where 
packet  loss  becomes  major  problem. 

Since  the  system  is  built  on  IP  network,  the  sensory  system  can  be  applied 
to  small  room,  as  applied  for  this  project,  to  very  large  scale  system.  By  intro¬ 
ducing  gateway  on  the  system  controller,  the  system  can  even  be  clustered  to 
form  hierachial  sensory  system. 

In  software,  all  sensors  are  categorized  into  static  and  active  sensor.  The 
operation  modes  of  the  sensors  are  simplified  to  surveillance  and  tracking  mode. 
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The  software  system  was  designed  in  UML  (Unified  Modeling  Language)  and 
object-oriented  software  design  technique  and  C-|— I-  language  was  used  to  im¬ 
plement  the  software  system. 

Four  functional  sensor  devices  and  system  controller  are  developed.  They 
are  actuated  three-dimensional  visual  tracking  sensor,  two-dimensional  audio 
object  sensor,  two-dimensional  multi  camera  visual  object  sensor,  and  satellite 
visual  sensor. 

In  the  following  chapter,  technical  details  of  the  developed  system  are  de¬ 
scribed. 
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Chapter  2 


System  Details 


2.1  System  Architecture 

The  sensory  system  is  built  on  IP  network.  All  sensors  are  connected  to  Ethernet 
through  network  controller  as  shown  in  figure  2.1.  Only  the  exception  is  the 
control  of  servo  motors  that  are  controlled  directly  from  controller  to  achieve 
low  latency.  Theoretically  they  could  be  controlled  through  IP  network  as  well. 


Figure  2.1:  Network  Structure  of  the  System 


All  modules  in  the  system  are  half  autonomous  in  so  that  the  telemetry 
data  are  reported  from  sensors  automatically  to  the  controller  but  all  controls 
are  commanded  from  the  controller  unidirectional. 

The  system  has  two  operating  modes,  survey  and  tracking.  In  the  surveil¬ 
lance  mode,  system  scans  through  the  area  to  find  any  moving  object  in  the 
area.  When  the  system  finds  a  suspicious  object,  the  mode  switches  to  tracking 
mode  to  track  and  record  suspicious  object. 

2.2  Sensor  Fusion  Processor 

Sensor  fusion  processor,  hereafter  called  system  controller  otherwise  noted,  re¬ 
ceives  telemetry  data  via  IP  network  and  commands  mode  transitions  and  motor 
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positions.  All  the  decisions  regarding  mode  transitions  are  made  in  system  con¬ 
troller. 

The  information  reported  from  sensors  are  suspicion  level  and  coordinate  of 
the  suspicious  object.  Information  from  sensors  are  prioritized  then  decisions 
are  made  upon  given  information  by  system  controller. 

2.2.1  Principle 

System  controller  has  following  priority: 

1.  Three  dimensional  vision  sensor 

2.  Two  dimensional  audio  object  sensor 

3.  Two  dimensional  vision  object  sensor 

At  the  development  stage  now,  satellite  vision  sensor  is  not  yet  integrated 
to  the  system  that  can  be  controlled  by  the  system  controller. 

Surveillance  Mode 

Surveillance  mode  detects  moving  objects  by  background  subtraction.  The  co¬ 
ordinate  of  the  moving  object  is  reported  to  the  system  controller.  The  system 
controller  behaves  as  shown  in  figure  2.2.  In  surveillance  mode,  the  controller 
actuates  the  motors  on  the  three  dimensional  vision  sensor  to  the  predefined 
positions  in  predefined  order  and  compares  the  aquired  image  with  the  tem¬ 
plate,  which  is  background  image  averaged  over  time.  When  moving  object  is 
detected,  the  system  moves  to  the  detected  position  and  tries  to  find  moving 
object  for  special  temporary  position  id=99.  If  the  system  finds  moving  object 
then  the  system  switches  to  tracking  mode  to  track  and  record  the  object.  If  it 
does  not  find  any  moving  object  then  the  system  recovers  to  the  preprogrammed 
sequence. 

Tracking  Mode 

In  tracking  mode,  the  system  tracks  the  object  with  the  three  dimensional  vision 
sensor. 

It  has  a  weighing  function  that  is  highest  in  screen  center  and  lowest  on 
edges,  so  the  sensor  tracks  easier  to  the  object  near  to  the  screen  center. 

Putting  the  weighing  function  w{x,y),  coordinate  of  the  object  {xi,yi),  and 
suspicion  level  of  t-th  object  Si,  evaluating  function  is  defined  as  follows. 

Qi  =  w{x„yi)  ■  s,  (2.1) 

An  object  which  maximizes  Oi  is  chosen  to  be  tracked.  Thus  weighing  function 
serves  as  a  fovea  which  makes  it  easier  to  concentrate  to  object  being  tracked 
when  multiple  objects  are  detected. 
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(  start  ) 


Figure  2.2:  Mode  Transition  of  the  System 


Let  the  inverse  kinematics  from  camera  position  and  screen  position  to  motor 
angle  f~^,  Sensor  actuation  algorithm  is  defined  as  follows. 

[0[k  + 1],  (/.[fc  + 1]]^  =  K  ■  (2.2) 

Here,  K  is  the  proportional  feedback  gain. 

The  system  has  geometrical  singular  point  at  (/)  =  90deg.  To  avoid  the 
singularity,  the  rule  9  =  0  when  (j)  =  OOdeg  was  used. 

2.2.2  Software 

The  system  controller  runs  on  Microsoft  Windows.  The  software  is  modular 
and  new  sensors  can  easily  be  added. 

A  structure  of  packet  being  transmitted  between  the  system  controller  and 
each  modules  is  shown  in  table  2.1. 

Table  2.1:  Packet  Structure 
Header  0 
Header  1 
Object  0 
Object  1 


Header  0  is  the  section  that  contains  the  summary  of  the  sensory  data. 
Typically  it  contains  which  sensor  the  packet  is  from,  size  of  the  packet,  and 
number  of  objects  detected.  There  are  some  more  additional  information  that 
is  sensor  dependent. 

Header  1  contains  commands  from  the  controller  to  the  sensors.  Now  it  is 
only  used  for  three  dimensional  visual  sensor,  since  only  this  sensor  has  tracking 
ability  currently. 

Number  of  object  section  depends  on  how  many  objects  were  detected.  It 
can  be  any  number  above  zero.  It  contains  suspicion  level  and  position  of  the 
detected  object. 

The  packet  is  transmitted  by  UDP  on  specified  port.  The  details  on  the 
packet  will  be  discussed  on  each  sensor. 

2.3  Three-Dimensional  Visual  Object  Tracking 

Shown  in  figure  2.3  is  the  three  dimensional  visual  object  tracking  sensor.  This 
sensor’s  functionality  are  as  follows: 

•  Acquire  three  dimensional  depth  map 

•  Detect  suspicious  object  by  subtracting  background  depth  map 
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•  Survey  with  preprogrammed  position 

•  Report  coordinates  and  degree  of  suspicion  to  controller 

•  Continuously  report  coordinates  of  suspicious  area  once  tracking  command 
is  issued  from  controller 

•  Receive  target  motor  position  from  controller  and  move  to  the  designated 
position 


Figure  2.3:  Three-Dimensional  Visual  Object  Tracking  Sensor 


Figure  2.4:  Three-Dimensional  Visual  Object  Tracking  Sensor  Block  Diagram 


2.3.1  Principle 

Three  dimensional  visual  sensing  was  achieved  by  stereo  vision.  The  reason 
for  using  three  dimensional  sensing  is  to  be  robust  to  the  change  in  ambient 
brightness  and  object  texture. 

Stereo  vision  is  based  on  range  finder  technique  and  detects  distance  by 
measuring  offset  in  right  and  left  image  (See  figure  2.5).  Cross  correlation  or 
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SAD  (Sum  of  Absolute  Difference)  is  used  to  find  the  corresponding  area  on  the 
image. 


Camera  L 


CC 

Camera  R 


®L1  ^L2 

,_1.A  1 

J  6r2 

OPI 


4 

Object 


Figure  2.5:  Principle  of  Stereo  Vision 


Three  dimensional  visual  object  sensing  has  following  advantages  over  two 
dimensional  object  sensing. 

•  Ambient  brightness  does  not  affect  the  sensing 

•  Sensitivity  does  not  change  between  colors 
However  it  has  following  disadvantages  at  the  same  time. 

1.  Requires  fast  processor 

2.  Recognition  is  disturbed  by  object  without  texture  or  texture  running  in 
the  direction  of  range  finder 

3.  Cameras  have  to  be  calibrated 

4.  Camera  frame  must  be  synchronized 

The  cameras  have  to  be  calibrated  to  get  equal  output  for  equal  images. 
Otherwise  correlation  of  the  image  would  be  inaccurate.  Camera  frame  must 
be  synchronized  for  the  same  reason. 


2.3.2  Optics 

To  resolve  the  item  three  and  four  of  disadvantage,  single  camera  was  used  to 
acquire  the  image.  Field  of  view  was  splitted  in  two  with  mirrors  as  shown  in 
figure  2.6.  By  splitting  the  image  in  two,  baseline  is  introduced.  After  capturing 


Left  View 

Right  View 


Mirrorl 


Camera 


Object 


Mirror2 


Figure  2.6:  Arrangement  of  Mirrors 


camera  image,  left  and  right  view  in  one  frame,  view  are  splitted  and  treated  as 
separate  image. 

Since  only  one  camera  is  used,  synchronization,  color  calibration,  and  bright¬ 
ness  control  is  not  necessary  as  in  conventional  stereo  vision  using  two  cameras. 

NTSC  analog  CCD  camera  was  used.  The  selected  device  is  equipped  with 
high-quality  optics  which  is  necessary  to  capture  detailed  texture  which  is  im¬ 
portant  in  calculating  accurate  distance.  The  image  was  captured  with  frame 
grabber  in  PC  with  Pentium  4  3GHz  with  hyper  threading.  High  performance 
computer  was  needed  to  calculate  depth  in  high  frames  rate  in  half  VGA  reso¬ 
lution. 

2.3.3  Mechanical  Structure 

The  sensor  is  equipped  with  a  pair  of  motors,  arranged  orthogonal,  making  it 
possible  to  track  detected  object. 


Figure  2.7:  Arrangement  of  Motors 


Pulse  commanded  AC  servo  motor  with  Harmonic  Drive  gear  head  was  used. 
Harmonic  Drive  reducer  was  necessary  to  eliminate  the  vibration  caused  by 
gear  backlash.  In  vision  systems,  vibration  with  amplitude  of  few  tenth  of 
degrees  can  easily  result  in  vibration  of  few  tens  of  pixel,  which  is  catastrophic 
in  constructing  vision  servo  system.  Command  for  motor  is  given  by  sensor 
fusion  processor  discussed  in  section  2.2.1. 

2.3.4  Software 

The  three  dimensional  recognition  engine  was  purchased  from  CyVerse  Corpo¬ 
ration  The  software  was  customized  in  following  ways: 

•  Returns  UDP  packet  in  specified  manner 

•  Detect  object  with  both  two  dimensional  and  three  dimensional  calcula¬ 
tion  and  reports  it  to  the  controller. 

^http:/ / www.cyverse.co.jp 
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Table  2.2:  Three  Dimensional  Video  Packet  Header  0  (Status) 


Field  Name 

Data  Size(Byte) 

Data  Type 

Description 

ID 

4 

4  Chars 

“VD”  +  null 

Packet  Size 

4 

Long 

Size  in  byte  including  this  header 

Mode 

4 

Long 

Survey/Tracking 

Preset  ID 

4 

Long 

Preset  ID 

Suspicion  Level 

4 

Long 

0-100% 

No.  of  Objects 

4 

Long 

Number  of  detected  objects 

Empty 

8 

N/A 

Table  2.3:  Three  Dimensional  Video  Packet  Header  1  (Command) 


Field  Name 

Data  Size(Byte) 

Data  Type 

Descripton 

Mode  Switch 

4 

Long 

-l:default, 

0:surveillance,  l:tracking 

Preset  Check 

4 

Long 

-l:default 

0-n:  Preset  ID  to  be  checked 

Init  Preset 

4 

Long 

-l:none 

0-n:  Preset  ID  to  be  initialized 

Empty 

20 

N/A 

2.4  Two-Dimensional  Audio  Object  Sensing 

Figure  2.8  shows  the  structure  of  two  dimensional  audio  object  sensor. 

This  sensor  has  following  functionality: 

•  Detection  of  sound  above  specified  level 

•  Eliminate  background  noise  by  adaptive  filtering 

•  Calculate  position  of  sound  source 

•  Report  position  of  sound  source  and  sound  level  to  the  controller 

2.4.1  Principle 

Sound  sensor  is  two  dimensional  microphone  array.  Position  of  the  sound  source 
is  defined  by  angle  of  source  as  shown  in  figure  2.10.  ^ 

^Technically  it  is  possible  to  calculate  the  coordinate  of  the  sound  source  in  x  and  y,  but 
considering  about  the  accuracy  of  the  localization  and  field  of  view  of  the  three  dimensional 
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Table  2.4:  Three  Dimensional  Video  Packet  Object 


Field  Name 

Data  Size(Byte) 

Data  Type 

Description 

Suspicion  Level 

4 

Float 

0-100% 

Position  X 

4 

Float 

±1.0  relative  position  from  screen  center 

Position  Y 

4 

Float 

±1.0  relative  position  from  screen  center 

Distance 

4 

Float 

meter 

Empty 

16 

N/A 

Figure  2.8:  Two-Dimensional  Audio  Object  Sensor 


Figure  2.9:  Two-Dimensional  Audio  Object  Sensor  Block  Diagram 


sound  source 


Figure  2.10:  Sound  Source  Position 


11 


Let  us  start  from  the  sound  source  identification  using  linear  microphone 
array  (See  figure  2.11).  When  a  sound  is  generated  at  point  A,  there  would 


Figure  2.11:  Linear  Microphone  Array 


be  a  phase  delay  in  sound  wave  among  microphone  array  due  to  the  difference 
in  distance  from  the  sound  source.  The  difference  distance  can  be  calculated 
from  the  phase  delay  and  speed  of  sound.  Assuming  the  “standard”  ambient 
condition,  20  degrees  Celsius  and  Ibar,  the  distance  can  be  calculated  with  some 
accuracy.  From  this  difference  in  the  distance,  position  of  the  sound  source  can 
be  localized. 

We  extended  this  idea  to  two  dimensional  microphone  array. 


Figure  2.12:  Two  Dimensional  Microphone  Array 


With  the  configuration  shown  in  figure  2.12,  angles  from  two  adjacent  mi¬ 
crophone  array  can  be  calculated  and  therefore  coordinate  of  the  sound  source 
can  be  localized. 


vision  sensor  (we  assumed  the  usage  with  vision  sensor  because  those  sensors  can  be  comple¬ 
mentary),  it  was  thought  to  be  reasonable  just  calculating  the  angle  of  the  source.  In  cases 
such  as  camera  mounted  five  meters  high  ceiling  with  room  size  over  20  X  20m?,  it  can  be 
reasonable  to  calculate  the  coordinate. 
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The  phase  difference  is  calculated  by  taking  cross  correlation  with  central  mi¬ 
crophone  every  256  milli-seconds.  Two  adjacent  microphone  arrays  are  chosen 
by  comparing  maximum  sound  intensity  of  the  microphone  array.  The  advan¬ 
tage  of  using  cross  correlation  is  that  the  signal  level  of  the  microphones  do  not 
have  to  be  adjusted  accurately.  Since  the  calculation  of  phase  difference  is  done 
in  time  domain,  theoretically  difference  in  signal  intensity  between  microphones 
does  not  affect  the  result.  However,  it  has  to  be  adjusted  to  the  level  that  it  can 
identify  the  position  of  the  sound  source  from  the  intensity  difference  to  decide 
which  two  adjacent  microphone  arrays  to  be  used. 

The  noise  filtering  is  done  by  eliminating  background  noise  bandwidth  af¬ 
ter  FFT’ing  the  captured  waveform  then  inverse  FFT  was  used  to  get  time 
domain  waveform.  This  method  was  chosen  due  to  the  hardware  support  of 
FFT  in  microprocessor  that  could  reduce  the  calculation  load  by  70%.  With 
microprocessor  with  vector  support,  FIR  may  also  be  useful. 

2.4.2  Audio  Sensor 

Twelve  microphones  are  used  in  the  microphone  array;  four  on  each  side,  sharing 
the  corner  microphone.  Microphones  are  connected  to  amplifier  to  adjust  signal 
level. 

All  of  the  microphone  signals  are  then  captured  with  A/D  converter  PCI 
card.  A/D  converter  card  with  simultaneous  capturing  capability  was  chosen  to 
guarantee  the  accuracy  of  phase  delay  detection. 

2.4.3  Mechanical  Structure 

Microphones  are  arranged  as  shown  in  figure  2.13. 


□  210 


•  • 


Figure  2.13:  Arrangement  of  the  Microphones 


They  are  fixed  in  formed  rubber  to  isolate  from  the  vibration  and  avoid  the 
reflection  from  the  fixation  surface.  As  mentioned  in  prior  section,  microphone 
array  was  chosen  as  a  complementary  sensor  to  the  three  dimensional  vision 
sensor.  Therefore  the  microphone  array  is  placed  around  the  three  dimensional 
vision  sensor. 

Purely  from  the  accuracy  point  of  view,  the  pitch  of  the  microphone  should 
be  as  long  as  possible  because  the  phase  delay  becomes  larger  that  makes  the 
measurement  more  accurate,  similarly,  number  of  the  microphones  should  be 
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as  much  as  possible  to  average  out  the  measurement  error.  However,  in  reality, 
the  size  of  the  array  is  limited  considering  the  usage  such  as  fixing  it  onto  the 
ceiling.  Also,  number  of  microphones  are  limited  by  the  capturing  channel  of  the 
capture  device.  The  pitch  of  the  microphone  was  chosen  from  the  calculation 
of  sampling  rate  of  the  capture  device  and  speed  of  sound  to  have  enough  time 
resolution.  Number  of  microphones  was  chosen  so  the  microphone  array  becomes 
axis  symmetric  and  still  less  than  the  number  of  channels  in  conventional  capture 
card  which  is  sixteen. 

2.4.4  Software 

The  software  was  developed  together  with  Harada  Laboratoy  in  Tokyo  Univer¬ 
sity  of  Science  The  sound  localization  was  done  on  PC  with  Pentium  4  2GHz. 
Intel’s  Math  Kernel  Library  was  used  to  make  use  of  microprocessor  dedicated 
commands  to  enhance  the  arithmetic  performance. 

Two  dimensional  audio  sensor  is  categorized  as  static  sensor.  It  reports  the 
coordinate  of  the  sound  and  sound  level  to  controller  in  following  packet  format. 


Table  2.5:  Audio  Packet  Header  0 


Field  Name 

Data  Size(Byte) 

Data  Type 

Description 

ID 

4 

4  Chars 

“AD”  -b  null 

Packet  Size 

4 

Long 

Size  in  byte  including  this  header 

Mode 

4 

Long 

N/A 

Preset  ID 

4 

Long 

N/A 

Suspicion  Level 

4 

Long 

0-100% 

No.  of  Objects 

4 

Long 

N/A 

Empty 

4 

N/A 

Table  2.6:  Audio  Packet  Header  1 


Field  Name 

Data  Size(Byte) 

Data  Type 

Description 

Empty 

32 

N/A 

2.5  Two-Dimensional  Visual  Object  Sensing 

Figure  2.14  shows  the  structure  of  two  dimensional  visual  object  sensor. 
This  sensor  has  following  functionality: 


®http:  / /www.te.noda.sut.ac.jp/pub/labs/harada/index-j.html 
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Table  2.7:  Audio  Packet  Object 


Field  Name 

Data  Size(Byte) 

Data  Type 

Description 

Suspicion  Level 

4 

Float 

0-100% 

Position 

4 

Float 

±180  deg 

Empty 

24 

N/A 

Camera  Switch  YUV  Image 


Camera 
and  LED 
Connectors 

Figure  2.14:  Two-Dimensional  Visual  Object  Sensor 


Figure  2.15:  Camera  Module 
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lOOBTX 


Figure  2.16:  Two-Dimensional  Visual  Object  Sensor  Block  Diagram 


•  Detection  of  motion  in  screen  by  background  subtraction 

•  Eliminate  background  fluctuation  by  adaptive  Altering 

•  Calculate  position  of  moving  object 

•  Report  position  of  moving  object  and  suspicion  level  to  the  controller 

2.5.1  Principle 

In  comparison  to  the  three  dimensional  vision  sensor,  two  dimensional  vision 
sensor  is  equipped  with  fixed  multiple  cameras  to  aquire  images  from  all  direc¬ 
tion. 

Since  the  cameras  don’t  move,  surveillance  can  be  done  with  least  power 
and  operation  can  be  done  without  being  know  by  people  who  broke  in.  Images 
from  multiple  cameras  are  aquired  by  time  sharing.  Sensor  controller  switches 
between  cameras  to  scan  through  the  room. 

Detection  of  the  suspicious  object  is  done  with  two  dimensional  image  com¬ 
parison.  Background  image  is  averaged  over  specified  time  to  reduce  error  re¬ 
duction  caused  by  change  in  brightness  or  lighting  condition.  Position  of  each 
cameras  must  be  preprogrammed  in  the  system  controller  for  the  system  con¬ 
troller  to  know  which  way  to  point  the  three  dimensional  vision  sensor  when 
switching  to  tracking  mode. 

VGA(640x480)  resolution  CMOS  image  sensor  was  used  as  the  camera  de¬ 
vice.  The  output  from  the  camera  is  parallel  8bit  YUV422  and  control  of  the 
device  is  done  through  I2C. 
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2.5.2  Software 


Table  2.8:  Two  Dimensional  Video  Packet  Header  0 


Field  Name 

Data  Size(Byte) 

Data  Type 

Description 

ID 

4 

4  Ghars 

“GA”  +  null 

Packet  Size 

4 

Long 

Size  in  byte  including  this  header 

Mode 

4 

Long 

N/A 

Preset  ID 

4 

Long 

N/A 

Suspicion  Level 

4 

Long 

0-100% 

No.  of  Objects 

4 

Long 

Number  of  detected  objects 

Empty 

4 

N/A 

Table  2.9:  Two  Dimensional  Video  Packet  Header  1 


Field  Name 

Data  Size(Byte) 

Data  Type 

Description 

Empty 

32 

N/A 

Table  2.10:  Two  Dimensional  Video  Packet  Object 


Field  Name 

Data  Size(Byte) 

Data  Type 

Description 

Suspicion  Level 

4 

Float 

0-100% 

Gamera  ID 

4 

Long 

ID  of  camera  detected  the  object 

Position  X 

4 

Float 

±1.0  relative  position  from  screen  center 

Position  Y 

4 

Float 

±1.0  relative  position  from  screen  center 

Empty 

16 

N/A 

2.6  Satellite  Visual  Sensor 

Figure  2.17  shows  the  structure  of  satellite  video  sensor. 
This  sensor  has  following  functionality: 

•  Streaming  of  Motion  JPEG  over  Ethernet 

•  Streaming  of  video  data  over  3G-324M  (video  phone) 
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Figure  2.17:  Satellite  Visual  Sensor 


lOOBTX  or  802.11b 


Figure  2.18:  Block  Diagram  of  the  Satellite  System 
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2.6.1  Principle 

As  seen  from  figure  2.18,  basic  operating  principle  is  similar  to  that  of  two 
dimensional  vision  sensor.  Except  in  satellite  sensor,  in  order  to  pack  all  the 
hardware  functionality  in  to  a  hemisphere  with  diameter  of  120mm,  hadware 
layout  is  very  different. 

Modules  connected  currently  are  only  image  devices,  but  the  architecture 
supports  other  sensors  such  as  PIR  sensors  and  temperature  sensors. 

Network  controller  captures  YUV  data  transmitted  from  cameras  and  en¬ 
codes  it  to  either  MPEG-4  or  Motion  JPEG.  Network  media  is  100  Base-TX  by 
default  but  IEEE  802.11b  wireless  Ian  and  video  phone  access  via  3G-324M  is 
also  supported. 

Unlike  other  sensor  modueles,  satellite  vision  sensor  can  be  operated  stand¬ 
alone.  Right  now,  object  detection  and  integration  to  the  system  are  not  im¬ 
plemented  on  this  module. 
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